Rules Cheat Sheet

Rules help you to take full advantage of Coralogix log parsing capabilities. For example, you can create your own rules to convert plain text logs into structured JSON logs or extract specific data from the log message as the value to a new JSON key. Coralogix parsing rules are applied prior to the indexing and storing of your logs, so you have full control over the final structure of your data.

1. Extract

You can extract any data within the log using an extract rule. You can extract the data into one of Coralogix metadata fields (severity, category, className, methodName, threadId) or extract that data into a new JSON key that will be added to your log. The rule will be applied on any log that matches our REGEX criteria and the original log text/JSON remains in full.

A common use case will be to extract the severity tag from the log payload, if exists, and override the log severity key when such was not mentioned in the log metadata. Another common action is to extract data from the log message in order to query the logs more easily. For example, extract a user ID that was sent as part of the message text into a new JSON field in order to perform range queries with it. You will be able to perform numeric aggregations once the user ID is stored in its own field.

Before the Rule is Applied

{
  "application": "prod",
  "subsystem": "platform",
  "status": 200,
  "message": "user login was successful user id=44028351"
}

REGEX

“message”\s*:\s*".*id=(?P<userId>[0-9]+)

After the Rule is Applied

{
  "application": "prod",
  "subsystem": "platform",
  "status": 200,
  "message": "user login was successful user id=44028351",
  "userId": "44028351"
}

2. Replace

You can replace any data within the log using a replace rule. For example, renaming a field name to fix a mapping exception in your logs or masking sensitive values such as credit card numbers, etc. ***To learn more about mapping exceptions, visit our post.

Example 1

Message as a string:

The request has succeeded.

Message as an object:

{
"message":  {
  "status_code": 200,
  "code_definition": "The request has succeeded."
   }
}

In Opensearch, the type of keys is set dynamically, sending two logs with the same field name and with different types of data (string ; object) will result in a race condition to the index mapping and message key type will be either string or object, depends which log came first. Eventually, all future logs with the opposite data types (for example, the type in the mapping of field message is string and logs are flowing with the message as object) will encounter a mapping exception.

Replace message with message_object will set the name of the key as "message_object" that will have its own mapping (object) in our index.

REGEX

"message"\\s\*:\\s\*{

Replace to

"message\_object": {

Result

{
  "message_object": {
    "status_code": 200,
    "code_definition": "The request has succeeded."
  }
}

Logs with a string value in the message will remain unchanged, the field name remains “message”.

Example 2

The log:

{
  "name":"John Smith",
  "credit_number":"1234-5678-1234-5678",
  "expiration_date":"01/01",
  "id_number":123456789
}

REGEX

(?P<mask>\d{4}-\d{4}-\d{4})-(?P<last_four>\d{4})

Replacement string

XXXX-XXXX-XXXX-${last_four}

Result

{
  "name":"John Smith",
  "credit_number":"XXXX-XXXX-XXXX-5678",
  "expiration_date":"01/01",
  "id_number":123456789
}

3. Block

Any log that matches a block rule’s REGEX will be blocked and won’t be indexed. In order to block a log, the REGEX should match any text provided in the log. *Note - Blocked logs are not counted in your daily quota.

Example

REGEX

”app_name”\s*:\s*”(web|mobile|IOS)”

The following log will get blocked:

{
  “app_name”:”IOS”,
  “subsystem_name”:”iPhone”,
  “timestamp”:1551774844957
}

4. Allow

With the usage of allow rules, you can filter your data and see only the logs that match your REGEX.

Example

You have these 4 logs:

{
  "applicationName":"web_entry",
  "ServerName":"eu-01",
  "category":"coralogix",
  "threadID":"u@G-QXEGeFz"
}

{
  "applicationName":"web_entry",
  "ServerName":"eu-02",
  "category":"coralogix",
  "threadID":"bp&67z#wPNY"
}

{
  "applicationName":"web_entry",
  "ServerName":"eu-03",
  "category":"coralogix",
  "threadID":"pYY8@ePae23H"
}

{
  "applicationName":"web_entry",
  "ServerName":"eu-04",
  "category":"coralogix",
  "threadID":"FWRGJqz8YEB"
}

REGEX

"ServerName"\s*:\s*"eu-(02|04)"

Only logs 2 and 4 will pass through our filter.

5. Json Extract

With JSON structured Logs, you can extract any field's content into one of Coralogix metadata fields (severity, category, className, methodName, threadId). A common use case is when you want to override the log severity, category, etc.

Example

In cases we have a JSON log message with a field mentioning the log severity while such was not assigned to the log metadata severity key (log severity will be set to 'DEBUG') we'll want to extract the severity from that field within the log to Coralogix metadata severity key.

{
  "message": {
    "severity": "Info",
    "code definition": "The request has succeeded."
  }
}

{
  "message": {
    "severity": "200",
    "code definition": "The request has succeeded."
  }
}

Source Field

JSON Key

Destination Field

Severity

Log Severity

Note that is possible to use a numerical field to set the severity of a log. The following table displays how errors will be displayed in your Coralogix UI.

Numerical field	Severity
2xx	info
4xx	error
5xx	critical

6. Parse

Using parse rules you can convert your unstructured log data (plain text) into JSON structure logs.

When sending JSON structured logs, Coralogix automatically parses them into JSON structured logs and they are mapped to Opensearch. You can read more about this here: https://coralogix.com/tutorials/auto-json-parsing/

Example

SeverityException in thread "main" java.lang.NullPointerException at com.example.myproject.Book.getTitle(Book.java:16) at com.example.myproject.Author.getBookTitles(Author.java:25) at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

REGEX

Exception(?: in thread) "(?P<threadName>[^"]+)" (?P<changethenamelater>[^\s]+)\s+(?P<stackeholder>.*)

Results

{
  "changethenamelater": "java.lang.NullPointerException",
  "stackeholder": "at com.example.myproject.Book.getTitle(Book.java:16) at com.example.myproject.Author.getBookTitles(Author.java:25) at com.example.myproject.Bootstrap.main(Bootstrap.java:14)",
  "threadName": "main"
}

7. Troubleshooting

Make sure you are creating the rule via the REGEX builder on the basis of several log examples.
Make sure you are taking a log example to test directly from the 'Explore Logs' view and copying the text from the 'View Raw Log' option.
Ask yourself, "Are our rules in the right order within the group? Are the groups in the right order?" Since rules are running group by group and within a group you'll escape the group once you get a match.
Block rules should be placed in their own group with no other rules.
Defaultly, rules are applied on the entire log text. If you want to apply it on a specific field within the log (with JSON logs), make sure that your source field in the rule is set correctly or keep it as 'Text' and in your REGEX enforce a match only when you match the field's name and its text, for example, "key":"YOUR_REGEX"

Log Parsing Rules

Next Auto JSON Parsing