Loggregation: Making Big Data Small

Loggregation© is an automatic log clustering feature that condenses millions of log entries into a narrow set of patterns using machine learning. It does so by automatically analyzing each log record sent to Coralogix, then separating the log constants from its variables.

Let’s have a look.

1) Below you can see, 1.7k log entries from a certain query:

2) Within seconds of clicking Templates, the 1.7k logs become grouped into their original patterns, thereby dramatically reducing the number of entries into 14 templates.

As you can see below, we went from 1.7k entries to only 14 templates. There is also a display of how many times each pattern arrived, the ratio of each pattern, and a full visualization of all parameters. This serves as a major timesaver in terms of log analysis.

3) Have JSON format log data? no problem! Coralogix clusters your JSON's into their unique appearances while allowing you to graphically view the different values including an automatic text clustering.

4) For further analysis, you can click on any of the variables and immediately display a graph with relevant information.

click on any of the variables and immediately display a graph with relevant information

5) To further drill into a template, hover it and click the magnifying glass icon to the right side of it.

In addition, clicking the pin icon at the top right corner of each visualization allows you to send that graph directly to your dashboard or Tags view in seconds:

Unclassified Logs

Unclassified logs (coralogix-unclassified-logs) are logs that do not belong to a specific template (coralogix_general_template) due to a specific reason. In this section we will go over those reasons and how you can correct them.

There are several main reasons that causes the system to not create templates. To see the specific reasons for each template branch - press on the "unclassified" button (see image above).

The reason you might see:

Cardinality of one of the fields is too high to generate a meaningful template
- Issue: In a specific template branch the cardinality of the values in one of the below fields is too high to create template. The system looks at one of the following fields ["text", "message", "msg", "log", "innerMessage"]. Should you have more than one of those fields in the log, the issue will resides with the first in the order that is listed above.
  Template branch details will be provided in the user interface.
- Possible mitigation: Using parsing to narrow the value in the relevant field to a shorter message.
Cardinality of the metadata exceeded the allowed amount
- Issue: One of the metadata fields contains high cardinality.
- Possible mitigation: Using parsing to reduce the cardinality. In case the values in one of the metadata fields is unique, please consider moving it to a different field.
Text too long
- Issue: The length of the log is too long to be able to create a template from it.
  Template branch details will be provided in the user interface.

When analyzing unclassified logs - please use the branch details to narrow down the issue.

branchDetails:{
"applicationName":"x",
"severity":"y",
"subsystemName":"z"
"Metadataxfield1":"xxx"
"Metadatafield2":"yyyy"}

Should you have any questions, please reach out to our support for further assistance.

Now you are ready to analyze a full day’s worth of data in a matter of minutes, including deep analysis of distributions and variables.
Save time and resources with Coralogix!

LiveTail

Next New Error and Critical Logs Anomaly