AWS Lambda Telemetry Exporter

This tutorial demonstrates how to set up and install the Coralogix AWS Lambda Telemetry Exporter.

This integration is one of two options - complete and basic - for monitoring Lambda, a requirement of our cutting-edge Serverless Monitoring feature.

This tutorial demonstrates how to set up Lambda monitoring to get basic telemetry, including logs, which requires setting up the Coralogix AWS Lambda Telemetry Exporter.
To set up Lambda monitoring to get complete telemetry, including traces, view the relevant documentation here.

Overview

The Coralogix AWS Lambda Telemetry Exporter is an AWS Lambda extension that uses AWS Lambda Telemetry API and seamlessly collects Lambda function logs, as well as Lambda platform logs, metrics, and traces. It is currently available as an open beta.

The Coralogix AWS Lambda Telemetry Exporter supersedes the previously offered Coralogix Extension for AWS Lambda (version 1.x.y) by extending its capabilities beyond logs. If you’re already using the Coralogix Extension for AWS Lambda, you can use it for now or migrate to the Telemetry Exporter to enjoy the new goodies early. We recommend using the Telemetry Exporter in new deployments.

This document assumes version 0.7.0 (layer version 29) of the Coralogix AWS Lambda Telemetry Exporter. If you’re using an older version, we highly recommend updating.

Setting up Lambda Monitoring

To monitor AWS Lambda functions using Coralogix, follow these steps.

STEP 1. Set up the Coralogix AWS Resource Metadata Collection.

STEP 2a. To get full telemetry, including tracing, set up the AWS Lambda Auto-Instrumentation. This is available for Node.js and Python (3.8 or newer). For other runtimes choose Step 2b.

STEP 2b. Alternatively, to get logs and basic telemetry, set up just the Coralogix AWS Lambda Telemetry Exporter

STEP 3. Learn what the Serverless Monitoring feature has to offer.

The document you’re viewing now is for Step 2b. You may want to proceed with Step 2a instead.

Prerequisites

An active AWS account with permissions to manage Lambda functions.
At least one AWS Lambda function selected for monitoring

Setup

The Coralogix AWS Lambda Telemetry Exporter is available with any of the following:

Using ARN of a lambda layer published by Coralogix (recommended)
Copying the telemetry exporter into a container image (recommended for Docker)
Deploying your copy of the lambda layer to your AWS account from AWS Serverless Repository

Installation: Published Layer ARN

If you need to deploy or update the Coralogix AWS Lambda Telemetry Exporter you can pick the ARN corresponding to the AWS region and CPU architecture from this list:

arn:aws:lambda:ap-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-north-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-north-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-west-3:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-west-3:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-west-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-west-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-west-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-west-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-northeast-3:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-northeast-3:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-northeast-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-northeast-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-northeast-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-northeast-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ca-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ca-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:sa-east-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:sa-east-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-southeast-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-southeast-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-southeast-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-southeast-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:us-east-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:us-east-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:us-east-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:us-east-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:us-west-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:us-west-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:us-west-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:us-west-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:af-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:af-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-east-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-east-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-southeast-3:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-southeast-3:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:me-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:me-south-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-south-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-south-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ap-southeast-4:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ap-southeast-4:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-central-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-central-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:eu-south-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:eu-south-2:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:me-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:me-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:il-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:il-central-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29
arn:aws:lambda:ca-west-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-x86_64:29
arn:aws:lambda:ca-west-1:625240141681:layer:coralogix-aws-lambda-telemetry-exporter-arm64:29

STEP 1. Configure the coralogix-aws-lambda-telemetry-exporter.

Access the Lambda function that you would like to monitor and configure the environment variables: Configuration > Environment variables
View the list of configuration variables in the Configuration section below.

STEP 2. Add the Telemetry Exporter layer to the chosen function.

Access the Lambda function from which you want to send telemetry data to Coralogix: Layers > Add Layer> Specify an ARN > paste ARN from the list provided above > Add

Notes:

If you are unsure about the architecture, it is displayed under Function runtime settings.

If that function was already using coralogix-lambda-extension, we recommend removing it: Layers > Edit > Select the Lambda layer > Remove

Installation: Container Image Lambda

Prior to your container image deployment, set up the Coralogix AWS Resource Metadata Collection.

To deploy your lambda as a container image and inject an extension as part of your function, build from the exporter image found here.

FROM coralogixrepo/coralogix-aws-lambda-telemetry-exporter:0.7.0 AS coralogix-extension
FROM public.ecr.aws/lambda/python:3.8
# Layer code
WORKDIR /opt
COPY --from=coralogix-extension /opt/ .
# Function code
WORKDIR /var/task
COPY app.py .
CMD ["app.lambda_handler"]

In this example, python3.8 is being used as the runtime, but all runtimes are supported with our lambda extension.

Notes:

There is no latest tag for the coralogix-aws-lambda-telemetry-exporter image; you must specify the version explicitly.
Find information on available versions/tags here.

Configuration

CX_DOMAIN: Coralogix domain within which you’ve set up your account.
CX_API_KEY: Coralogix Send Your Data - API Key.
CX_SECRET: As an alternative to providing the Send Your Data – API Key directly in CX_API_KEY you can use AWS Secrets Manager to store the key, and provide the name of the secret in CX_SECRET. See the Secrets Manager section below for more details.
CX_REPORTING_STRATEGY: Acceptable values include LOW_OVERHEAD, REPORT_AFTER_INVOCATION and REPORT_DURING_AND_AFTER_INVOCATION. For info on how to choose the optimal strategy for your Lambda function, see the Reporting Strategies section below.
CX_APPLICATION & CX_SUBSYSTEM (optional): Defaults to AWS account number and Lambda function, respectively. Learn more about application and subsystem names on your Coralogix dashboard here.
CX_FIREHOSE (optional): Can be used instead of CX_DOMAIN to deliver telemetry via Firehose. See Telemetry delivery via Firehose below.
CX_LOG_METADATA_ENABLED (optional): Defaults to true. Set to false to exclude the cx_metadata from logs in order to reduce their size.
CX_OTLP_SERVER_ENABLED (optional): Defaults to true. Set to false to disable the OTLP server and free up the 4317 and 4318 ports.
CX_LOG_ONLY (optional): When set to true disables metrics and tracing.
CX_LOG_MODE (optional): Defaults to structured. Acceptable values are:
- disabled - no logs will be delivered to Coralogix
- structured - logs will be wrapped in a JSON and delivered to Coralogix
CX_METRICS_MODE (optional): Defaults to platform_report. Acceptable values are:
- disabled - no metrics will be delivered to Coralogix
- platform_report - metrics based on AWS Telemetry API Platform Report event will be delivered to Coralogix
CX_OTEL_METRICS_MODE (optional): Defaults to processed. Acceptable values are:
- disabled - no metrics will be delivered to coralogix
- direct - OTEL metrics are sent as produced by OTEL SDK
- processed - OTEL metrics are processed to remove samples that don’t introduce an update and add samples, so that rate calculations are correct
CX_TRACING_MODE (optional): Defaults to otel when OTEL instrumentation is detected. Otherwise defaults to telemetry_api. Acceptable values are:
- disabled - no traces will be delivered to Coralogix
- telemetry_api - will generate spans based on data provided by the AWS Lambda Telemetry API.
- otel - will expect to receive spans from the function via OTLP. This mode is meant to be used in conjunction with OTEL instrumentation of the function and the CX_OTLP_SERVER_ENABLED enabled. Warning: otel tracing mode delays emitting logs until the end of the invocation even when the reporting strategy is set to REPORT_DURING_AND_AFTER_INVOCATION .
CX_TRACE_SAMPLING_MODE (optional): Defaults to all. Acceptable values are:
- all - all traces are sent to Coralogix
- follow_xray - the telemetry exporter will respect AWS XRay’s decision which traces should be sampled
CX_TAGS_ENABLED (optional): Defaults to false. Setting it to true will enable enrichment of the telemetry with the tags of the Lambda function. See the Enrichment with tags section to learn how to correctly set this up.
CX_TAG_CACHE_VALIDITY_MS (optional): Defaults to 300000 (5 minutes). Controls how often the tags will be refreshed.
CX_MESSAGE_SIZE_LIMIT (optional): Defaults to 30000. Log messages longer than CX_MESSAGE_SIZE_LIMIT bytes will be truncated.
CX_EXCLUDED_SPAN_ATTRIBUTES (optional): Defaults to disabled. Acceptable values are:
- disabled
- predefined - will remove known vulnerable attributes. Currently http.request.header.x-api-key and http.request.header.authorization. More may be added in the future.
- Any regex that will match attributes should be removed. For example: http\\.request\\.header\\.x-api-key|http\\.request\\.header\\.authorization
CX_REPORTING_DELAY_MS (optional): Defaults to 15000. Controls how often telemetry is sent in LOW_OVERHEAD and REPORT_DURING_AND_AFTER_INVOCATION modes
CX_MAX_SHUTDOWN_FLUSH_DELAY_MS (optional): Defaults to 600. Controls how long telemetry exporter will wait for the remaining telemetry to arrive during shutdown.
CX_SPAN_SENDING_THRESHOLD (optional): Defaults to 2048. Telemetry exporter will send function spans early if the number of spans exceeds this threshold.

Reporting strategies

Due to the serverless nature of AWS Lambda and Lambda extensions, the telemetry-exporter cannot freely do its job after and between invocations of the monitored Lambda function. A tradeoff has to be made between timely delivery of the telemetry and keeping overhead costs low.

The optimal strategy choice depends on how often the Lambda function is invoked and how long it runs. The coralogix-aws-lambda-telemetry-exporter offers three reporting strategies that enable you to adjust reporting to the characteristics of your function.

LOW_OVERHEAD : Recommended for frequently called functions. Telemetry is batched across many invocations of the function, and the telemetry-exporter avoids keeping the Lambda execution environment active after an invocation is complete (and in turn avoids additional billable time to the invocations). This may result in long delays (in the order of minutes) in the delivery of telemetry.
REPORT_AFTER_INVOCATION : Recommended for rarely called functions with short to moderate execution times. Telemetry is reported at the end of each invocation, limiting the amount of billable time for each invocation.
REPORT_DURING_AND_AFTER_INVOCATION : Recommended for rarely called functions with long execution times (15s or more). Telemetry is reported in regular intervals during the execution of the function and then after the execution completes. This strategy will add 2-3s of billable time to each invocation.

Notes:

Regardless of the chosen reporting strategy, technical limitations may place constraints on delivery times for some telemetry data.
Rather than being delivered directly after the invocation, some data (such as the total billable time of an invocation) is delivered during the next invocation of the Lambda function instance.

Using Metrics

Check out our tutorial here.

Enrichment with tags

Coralogix AWS Lambda Telemetry Exporter can read the tags of the AWS Lambda function and add them to all reported telemetry (logs/metrics/traces). This feature is disabled by default and can be enabled with CX_TAGS_ENABLED, but it requires prior configuration of permissions for the Lambda function. The configuration can be done in the following steps:

STEP 1. In AWS Console access the Lambda function that you would like to configure and navigate to: Configuration > Permissions

STEP 2. Click on the name of the execution role. This will bring to the IAM configuration of that role.

STEP 3. Click Add Permissions > Create Inline Policy

STEP 4. Switch to JSON view and paste the following policy JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "lambda:GetFunction",
            "Resource": "<paste lambda ARN here>"
        }
    ]
}

STEP 5. Replace the <paste lambda ARN here> with the ARN of the Lambda function. (you can find it on the main page of the function’s configuration)

STEP 6. Click Review Policy

STEP 7. Name the policy, for example ListLambdaTags.

STEP 8. Click Create Policy.

STEP 9. Set the CX_TAGS_ENABLED environment variable to true.

PrivateLink

The Coralogix AWS Lambda Telemetry Exporter can send telemetry to Coralogix over AWS Private Link by implementing the following steps:

STEP 1. Set up PrivateLink for your VPC.

STEP 2. Configure the lambda function to run in that VPC.

STEP 3. Change the CX_DOMAIN environment variable, by adding private. at the beginning. For example, if your CX_DOMAIN used to be coralogix.com , now it should appear as private.coralogix.com.

Code signing

The Coralogix AWS Lambda Telemetry Exporter Lambda layers are signed with arn:aws:signer:eu-west-1:625240141681:/signing-profiles/coralogix_lambda AWS Signer profile. You may need to add this profile to your code signing configuration if you have configured Lambda to require signed code.

Secrets Manager

Coralogix AWS Lambda Telemetry Exporter can retrieve the Send Your Data – API Key from AWS Secrets Manager. It requires prior configuration of permissions for the Lambda function. The configuration can be done in the following steps:

Create a secret with the Send-Your-Data API key in AWS Secrets Manager. You can lear how to create Secrets Manager secrets from this instruction https://docs.aws.amazon.com/secretsmanager/latest/userguide/hardcoded.html#hardcoded_step-1
In AWS Console access the Lambda function that you would like to configure and go to: Configuration > Permissions
Click on the name of the execution role. This will bring to the IAM configuration of that role.
Click Add Permissions > Create inline policy
Switch to JSON view and paste the following policy JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "<paste ARN of the secret>"
        }
    ]
}

Replace the <paste ARN of the secret> with the ARN of the Secret.
Click Review Policy
Give it a name. For example GetSecretValue
Click Create Policy
Set the CX_SECRET environment variable to the name of the secret
Remove the CX_API_KEY environment variable if you added it before

Telemetry delivery via Firehose

Lambda telemetry exporter can send telemetry to Coralogix via AWS Firehose Delivery Stream instead of using the Coralogix ingestion API directly.

Advantages of that approach are:

No direct route from Lambda to Coralogix is needed (it works for Lambdas in an isolated VPC)
Decoupling - Introducing Firehose as a middle man decouples lambda functions from Coralogix APIs. Errors, and variation in response times of Coralogix ingestion API no longer affect the functions.
High reliability - Firehose can wait for responses longer and retry more times, which makes delivery more reliable
Lower latency and cost - Latency from function to Firehose located in the same AWS region and account is low (~20ms) compared to latency of a call to Coralogix ingestion API (~130ms + cross region latency). This translates in shorter lambda executions, resulting in lower lambda costs. The difference is especially significant when Coralogix instance is on one continent, while lambda functions are running on other continents.

The disadvantages of using Firehose are:

Extra setup - you need deploy a Firehose delivery stream in each AWS region and give the functions permissions to write to that Firehose
Cost - AWS will charge the customer per GB of data sent though the Firehose. Keep in mind that this extra cost may be offset by the savings thanks to shorter lambda invocation duration.

Setup

STEP 1. Configure Firehose delivery stream configured like described here except with integrationType=CxOtlp.

STEP 2. Modify the execution role of the Lambda function. Give it permission to access the Firehose (you can use the AWS managed AmazonKinesisFirehoseFullAccess policy or fine tune your own policy).

STEP 3. Configure lambda-telemetry-exporter with CX_FIREHOSE set to the name (not ARN) of the Firehose delivery stream. Neither CX_DOMAIN nor CX_OTLP_URL should be set. CX_API_KEY shouldn’t be set either.

STEP 4. Make sure that the quota of put operations per second of the Firehose is higher than the expected peek number of lambda invocations. For any large Lambda deployment, that means that you should request a quota increase from AWS.

STEP 5. If using Lambda functions in an isolated VPC, configure a VPC endpoint for Firehose.

Update Coralogix AWS Lambda Telemetry Exporter

Update the procedure for ARN-based installations

STEP 1. Look up the latest layer version by reviewing the list of ARNs presented earlier. The number at the end of the ARN is the layer version.

STEP 2. Change the Telemetry Exporter layer version used by the function: Layers > Edit > Change to the updated layer version > Save

Update the procedure for AWS serverless repository-based installations

STEP 1. Deploy the Coralogix-AWS-Lambda-Telemetry-Exporter in your AWS Serverless Application Repository. You are required to deploy it in the same account and region as your Lambda function.

STEP 2. Verify the deployment by clicking on ‘Layers’ in the left-hand menu. You should see that the two previously deployed Lambda layers have new versions. Keep in mind that the version number of the Lambda layer is simply incremented with each deployment and does not correspond to the semantic version shown in the AWS Serverless repository.

STEP 3. Change the Telemetry Exporter layer version used by the function: Layers > Edit > Change to latest available layer version > Save.

Additional resources

Documentation

AWS Lambda Auto Instrumentation
Serverless Monitoring

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

AWS CloudWatch Metrics Processing

Next AWS Eventbridge