Custom Enrichment

Enrich log data with additional business, operational, or security information to enhance log analysis and utility.

Overview

Custom Enrichment allows you to enhance your logs by adding critical contextual data that may not be available at runtime. This enrichment is done by appending fields to your JSON logs based on specific matches using a custom data source you define.

There are two methods for enriching your logs:

1. Automatic enrichment during ingestion

2. Dynamic enrichment using DataPrime syntax

Automatic enrichment

Automatic enrichment occurs as logs are ingested. This process involves selecting a log key, using the key value to look up additional information, and automatically appending enriched fields to the logs.

Automatic enrichment ensures that logs are consistently enriched during ingestion, making the enriched data always included and readily available. This consistency simplifies log consumption through queries and ensures compatibility with third-party products that read from S3 buckets. Additionally, enriched fields can be used to define alert conditions in Alerts and to display statistics about enriched fields in widgets within Custom Dashboards.

DataPrime query enrichment

This method utilizes the DataPrime enrich query to enrich logs dynamically for the purpose of the query. This allows for the enrichment of previously ingested logs in Coralogix without increasing the stored log size, as the enrichment occurs dynamically only for the query results. Get started with our DataPrime Cheat Sheet.

How can I use Custom Enrichment?

Here are some use cases illustrating how Custom Enrichment can benefit you:

Monitoring

Gain visibility into customer names based on customer IDs.

Suppose you have a log with a UUID representing a customer but no field containing the customer's name. You can visualize and search logs based on this information by including a field with the customer's name. With Custom Enrichment, set up a CSV file mapping each UUID to a customer name, enriching the logs automatically as they are ingested.

Security

Receive alerts when users browse domains outside of your whitelist.

Consider a log field with a domain name representing where the user is browsing. You want to create an alert for any attempts to access your application from unauthorized domains. By setting up a CSV file with a list of whitelisted domains, you can enrich logs with a field containing the word "allowed" for each whitelisted domain. Then, create an alert for logs not containing this field, using a query such as NOT domain_enriched:allowed.

Required permissions

Users with the following permissions may view and/or manage your Custom Enrichment configuration or data.

Prepare a CSV file

Create a string-to-string or string-to-JSON CSV file that defines your custom data source. Once created, upload it to your computer.

String-to-string

For string-to-string, the file should include two columns:

• Column A will include a list of values to match.

• Column B will include the enrichment information. (The file must include headers, although column names have no specific meaning.)

Example of a string-to-string CSV file that contains pairs of UUIDs and customer names, enhancing readability for business and support teams:

String-to-JSON

For string-to-JSON, the file should include at least three columns:

• Column A will include a list of values to match.

• Columns B, C, etc., will include the enriched JSON object fields’ information. (The file must include headers, which will then be the enriched JSON field names correspondingly.)

Example of a string-to-JSON CSV file that contains in each row a UUID, customer name, and customer size:

Configuration

STEP 1. From your Coralogix toolbar, navigate to Data Flow > Data Enrichment > Custom Enrichment. Click + Add Custom Enrichment.

STEP 2. Configure the Custom Enrichment. Provide the enrichment name and description, then upload the CSV file you previously created.

STEP 3. [Optional] For automatic enrichment, define the enrichment during log ingestion. To enrich your logs on demand post-ingestion and avoid increasing log storage size, skip this section and use the DataPrime enrich command. Note that those opting for automatic enrichment may also use the DataPrime enrich command on demand, even though they have defined both keys for enrichment.

• Key. Select one or more keys for enrichment with + Add key.

• Enriched key name. Define the name of the enriched log key to structure the enriched log according to your needs rather than using the default <key>_enriched.

STEP 4. Click Create enrichment.

Example

Assume logs reporting cloud activities identify a user by their user ID who performed a certain suspicious activity. You can use Custom Enrichment to understand who this user is and assess whether the reported cloud activity is legitimate for that user or not.

Original log

{
  "action": "DeleteFile",
  "user_id": "1234"
}

Enriched log with string-to-string mapping

Enriching the log with the user's name based on the user ID gives you immediate context about who performed the action.

{
  "action": "DeleteFile",
  "user_id": "1234",
  "user_id_enriched": "John White"
}

Enriched log with string-to-JSON mapping

For a more comprehensive context, you can enrich the log with additional user information, such as their name, role, and department.

{
  "action": "DeleteFile",
  "user_id": "1234",
  "user_id_enriched": {
    "name": "John White",
    "role": "DevOps Engineer",
    "department": "IT"
  }
}

With this enriched data, you can better evaluate whether John White, a DevOps Engineer from the IT department, has a legitimate reason to perform the "DeleteFile" action. This will enhance your ability to identify suspicious activities and respond appropriately.

Limitations

CSVs uploaded for Custom Enrichment are limited to 150,000 rows. In order to ensure optimal performance for your log ingestion, CSVs with more than 10,000 rows may be used only for DataPrime query enrichment and not automatic enrichment upon ingestion.

Customers are limited to 8 Data Enrichment keys, which may combine Custom Enrichment keys and other keys.

API support

View our Custom Enrichment API documentation to create a Custom Enrichment via API.

Additional resources

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by emailing [email protected].

Previous AWS Resource Enrichment

Next Geo Enrichment

Theme

Light

light

dark