Log Normalization

Make data analysis easier with log normalization.

Overview

Managing and analyzing logs across various products presents a significant challenge due to the diverse structures and formats in which logs are stored. This diversity complicates searching for specific information across product logs, as each product may use different keys to store the same information. For example, the source IP address might be recorded under sourceIPAddress, source.ip, srcaddr, or other keys, depending on the reporting product. This discrepancy makes data comparison and aggregation difficult, resulting in complex and inefficient data analysis processes.

Log normalization simplifies data analysis by giving standard names to common values in logs and organizing them using parsing rules.

How It Works

Deploy one of many Coralogix extension packages to add the cx_security key to your product logs. This will normalize email addresses, IP addresses, and URLs across your logs into a single key name, even if the same data comes from different key names.

Keys created under cx_security include source_ip, destination_ip, email, url, domain, username, userAgent, event_name, service_name, result, source_hostname, destination_hostname, resource, event_result, event_name, target_email, target_username. Additional keys will be added from time to time.

Here are some examples:

{
   cx_security: {
      email: "[email protected]"
      source_ip: "1.1.1.1"
      destination_ip: "2.2.2.2"
      url: "www.google.com"
   }
   sourceIPAddress: "1.1.1.1"
   destinationIPAddress: "2.2.2.2"
   website: "www.google.com"
   user: {
      id: "[email protected]"
   }
}


{
   cx_security: {
      source_ip: "1.1.1.1"
      destination_ip: "2.2.2.2"
   }
   network: {
      src: "1.1.1.1"
      dst: "2.2.2.2"
   }
}

Getting Started

Get started with log normalization by deploying a Coralogix extension package. Once deployed, the log normalization parsing rules are automatically activated using parsing rules.

STEP 1. Navigate to Data Flow > Extensions from your Coralogix toolbar.

STEP 2. Choose an extension package bound to your integration. Select from any of our extension packages that include log normalization parsing rules:

1Password
Amazon Route53 DNS Query Resolver
Amazon VPC Flow Logs
Auth0
AWS ALB
AWS CloudTrail
AWS VPN
AWS WAF
Azure Audit
Cloudflare
CrowdStrike
CyberArk PAM
DUO Security
Fortigate
Fortigate Firewall
GCP Audit
GCP Security Command Center
GitHub Enterprise
Google Workspace
Google Workspace Alert Center
InfoBlox DNS
JumpCloud
Okta
Palo Alto
Prisma SASE
SentinelOne
Trend Micro AV
TrendMicro
UpGuard
Zscaler ZIA
Zscaler ZPA

Note that this list may not reflect the most updated list of extensions.

STEP 3. Define the parameters of the extension.

Select applications and subsystems for which alerts and parsing rules will be applied. These should be identical to those in the associated integration package.
If you have reached your parsing rules limit, please contact Customer Support to increase it.

STEP 4. Deploy.

Using Log Normalization

Explore Screen

Utilize the Explore screen to execute queries based on cx_security keys to retrieve all logs sharing the same value. For example, run the following to retrieve all activity logs associated with the user’s email [email protected]:

cx_security.email:”[email protected]”

Custom Dashboards

In your Custom Dashboard, create widgets grouped by normalized keys under cx_security.

For instance, create a horizontal bar chart grouping by cx_security.source_ip to visualize the top source IPs across product logs.

Alerts

Set up Flow Alerts to inform you know when a series of alerts with the same details happen in a row within a certain time. Grouping normalized keys under cx_security allows you to configure your Flow Alert with base alerts sharing the same value for these normalized keys.

For example, to detect a suspicious sequence of actions taken by the same user, where each action is reported by a different integration or product, create a Flow Alert where every alert has the same user's email in cx_security.email.

Enrichment

Select the normalized keys to be enriched by Unified Threat Intelligence and Geo enrichments, rather than selecting each of the original keys per product log.

For instance, select the cx_security.url key for the Unified Threat Intelligence enrichment and define an alert to detect the presence of the enriched cx_security.url_suspected key across all normalized product logs containing a URL.ֿ

Best Practices

We recommend log normalization in parallel to deploying Google Workspace Users, which adds user context to the normalized log keys.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to contact us via our in-app chat or by emailing [email protected].

Parse JSON Field

Next Dynamic Blocking