This Splunk app was designed as a troubleshooting tool and monitoring aid for Cribl administrators. It was created by Cribl Support Engineers to help customers troubleshoot their own Cribl deployments. There are several troubleshooting dashboards tailored to certain product areas in which support has seen the highest number of recurring issues. And while our intent is to help you troubleshoot your own Cribl deployment, this app will always be a continuous ”work in progress” and should always be used in conjunction with the Cribl monitoring console and associated views.
In the ancient art of troubleshooting, context is key - whether the problem is of a technical nature, or merely one related to existence itself.
Without a clear understanding of the circumstances surrounding an issue, it becomes challenging to identify the root cause and provide an effective solution. Context provides valuable information about a specific environment, and every environment is unique; when using this app, it is wise to ask yourself the following questions:
- What is the current issue you are attempting to troubleshoot?
- Are there any recent configuration changes that were made before the issue started?
- Were there any specific user interactions that may have contributed to the start of the issue? Example: Increase in data throughput, new sources or destinations, change in architecture, etc..
Answers to the above questions, and many others, will help narrow down the scope of the investigation, enabling you and your team to focus their efforts on the relevant areas. Additionally, context aids in replicating the problem, as it enables Support Engineers to understand the exact conditions under which the issue occurs. Knowledge of the environment, along with all of the context on use-cases and integrations, and ensure that the troubleshooting process is efficient and accurate.
This app and its associated dashboards rely on a few components which you must configure. Note: v.2.0.0 is intended only for Cribl Stream Worker nodes and Leaders. Future releases will add dashboards and functionality for Cribl Edge deployment troubleshooting and monitoring.
Cribl internal logs and metrics must be enabled and forwarded to Splunk in order for all of the panels to populate with events. Refer the documentation here for instructions on configuring this source. Be sure to configure a corresponding index and sourcetype field for logs and a corresponding index for metrics.
After installing or upgrading the CriblVision application, run the Populate Cribl Stream Worker Lookup report to repopulate the cribl_stream_workers lookup. This can be done by selecting "Populate Cribl Stream Worker Lookup" from the navigation menu. This report is scheduled to run every hour, but can be updated to meet your requirements.
This app ships with 4 macros which must be edited in accordance with your splunk index naming schema. For more info on configuring macros, reference Splunk’s Documentation here.
| Set_cribl_internal_log_index | Set this macro definition to the index you configured cribl logs. |
| Set_cribl_log_sourcetype | Set this macro definition to the sourcetype you configured for cribl logs. |
| Set_cribl_metrics_index | Macro definition for the index you configured for cribl metrics |
| Set_cribl_metrics_prefix(1) | change the "cribl.logstream" value before .$metric_name$ if you have changed the default namespace used for metrics on your cribl metrics internal source. |
Some of the views in this app will require Leader logs to be forwarded on to Splunk. In distributed Cribl Stream environments, Leader logs are currently NOT sent via our internal source. You will have to install an Edge node on your Leader node and configure local log collection via a file monitor input. Configure the file monitor input to collect logs by configuring the filename allow list modal to /opt/cribl/log/*.log. For more information on how to deploy an Edge node, please refer to our documentation here. Note: When deploying the Edge node to your Leader node, we recommend having a separate fleet just for this node. Be sure to disable all other inputs on that Edge node except for file monitor inputs.
In addition to this overview page, this app provides several views intended to aid a Cribl admin in troubleshooting and assessing the health of a Cribl deployment. Every view is equipped with a "how to use" button that reveals a description and instructions for that view when clicked. We recommend starting with the health check view and selecting a single worker node from the host dropdown.
Author: Johan Woger
Co-Authors: Jeremy Prescott, Martin Prado, David Sheridan, Christopher Owen
Honorable Mentions: George (Trey) Haraksin - For his initial ideas on thruput introspection (check out his other projects at https://github.com/arcsector)
Ben Marcus - General Testing.
Brendan Dalpe - Guru of many things.
Brandon McCombs - General Testing.
Chris Owens - General testing and contributor.