Unable to see ingested logs on console

Dear Community,

We are unable to see ingested logs on DNIF console.

We on boarded two devices(windows & Cisco ASA).

Collection status showing both are active.

On streams, we can see spike in numbers, but we are unable to find how to check those logs.

We used filter stream=other on find section, but the search never completes.

Also we see message - report server and correlation server down on management components section.

Kindly assist us how to check ingested data on console.

Thanks in advance.

Regards,
Satish

Hey @skollipara

It seems some services have not spawned correctly. Can you briefly mention how does your setup look like? Have you installed multiple components on same node/s?

Can you check and share snippets as mentioned here: Observing Error While Searching - #4 by Blackbird2Raven

1 Like

Hi,

Our setup having three below components.
1)Core & Console
2)Data Node
3)Adaptor

I am sharing requested snippets, Kindly check and suggest.
image|690x293





Regards,
Satish

Hey @skollipara

Can you restart Compute-Leader service and check again? → Troubleshooting Guide - Query Server Down

Once restarted, please validate the query and it’s execution as per the guide → Troubleshoot and diagnoze search execution

1 Like

Hi,

I have restarted the compute-Leader service, but no luck.

Also tried DQL command, it is neither showing any error nor any results.

Kindly help us to resolve.

Regards,
Satish

Did you get the query output after some time or few minutes later?


Can you share the snippets of steps in above guide, namely,

  1. Check the status of the compute-leader service on core
  2. Check the status of the compute service on the Datanodes.

PS: Start all stopped service for the specific components if you stopped.

Assuming Minimum Requirements and Prerequisites are already reviewed and met.

If despite that - the search never completes, it might be an indication of an issue with the internal analytics workers. This usually resolves with a simple Compute-Leader restart. Have you tried running the query again after sometime?

Could you also indicate what version of DNIF you’re running? (It’s visible at the bottom of the login screen.)

1 Like

Hi Nikhil,

Thank you.

The issue has been sorted out after increasing data node vCPU to 32 and achieving Hostname Resolution by using hosts file on all DNIF components.

But there is an issue of searching latest logs(Data is not available for selected duration).

We can search logs older than 4 hours data using visual filters.

also when we search DQL query we are getting Data is not available for selected duration.

.

Kindly let us know how to check if any backlogs in DNIF components as we observe no latest logs.

Regards,
Satish

1 Like

Hi @skollipara - With visual filter is it visible for the duration you need? Are you sure you have data ingested for the duration?

Which version you using?

Hi John,

We are using version 9.0.2
image

Still we are unable to search latest logs, last searchable logs were of 4 May’2021 5:21 PM(IST).

Also checked, all services are in green state.

Regards,
Satish

Hi Satish,

Data not available for the selected duration” indicates that specific time interval logs are not available in the system.

To verify, navigate to the Manage Components tab and in Adapter, you will be able to see the EPS timeline chart.

On the EPS Timeline chart if the EPS is 0, this indicates that the data isn’t ingested in the system, please check the configuration of the log source end.

If the EPS value > 0 indicates that the data is received on the Adapter and is visible on the Adapter EPS timeline chart, you can check the types of log data being ingested to the system using navigate to the Managing Streams section.

Also, you can check the Collection status page present under the System tab, it will display the list of all active log sources/devices.

Check if any error messages are visible under Notable Events, this will display errors/failures in your environment.

I think this can help us understand the problem properly.

Thanks,
Manish

Hi Manish,

I have checked the EPS timeline chart, EPS value >0.

I have checked the collection status, below are devices in active state.

I didn’t find any notable events of any errors recently.

Regards,
Satish

Hi Satish,

In the above snippet, it seems that the indexing rate chart is not updating, and it indicates that the data is not getting indexed in the system.

As per observation, there is something has been stalled at the service layer. Check the status of the Indexer Process service on the Adapter and restart it - View the Adapter services

Once restarted, you can validate the query and its execution as per the guide - Troubleshoot search

Thanks,
Manish

Hi Manish,

As suggested, I restarted indexer process, but still searches are not working.

Regards,
Satish

Dear Community,

All services are in green state and no error messages on notable events.

Devices are in green state and EPS is also showing good.

But still we are unable to search the latest logs.

Kindly let me know how to resolve the issue.

Thanks in advance.

Regards,
Satish

Hi Satish,

I assume you get the “Indexer has stopped” message in notable events after restarting the indexer process.

In addition, please ensure that all DNIF components and your desktop clock must be synchronized with a real-time or local/public NTP server.

To understand the overall problem statement, please reshare the below screenshot with us (do not crop the snippet, as the full-screen screenshot helps us to understand the problem properly).

  1. All components notable events
  2. Manage components graph
  3. Each components graph
  4. All components services status
  5. Collection status
  6. Streams data
  7. Last log received screenshot
  8. Version details

Thanks,
Manish

Hi there Satish - part of our community members are found at the DNIF discord channel. Join in, we can go over the issue once again.

Note that the measure in the Indexing Rate chart is MB/minute. Considering the low EPS of 20, the compacted data is probably a very small fraction of an MB hence being seen as zero.

Edit: corrected. misread the chat, I assumed the Indexing rate chart was being referred here.

1 Like

Hi Manish,

Thank you for the udpate.

I observe time difference on DNIF components.

I will update once the time difference issue is sorted.

Regards,
Satish

Hello Manish,

I didn’t observe the “indexer has stopped” message in notable events after restarting the indexer process.

Now all DNIF components are in same time zone(IST)

image

Regards,
Satish