AWS CloudWatch Metric Streams with Amazon Data Firehose

Amazon Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix, an AWS Partner Network (APN) Advanced Technology Partner.

Use this seamless integration for easy monitoring, flexibility with minimum maintenance, and maximum scalability. There is no limit on the number of delivery streams, so it can be used for getting data from multiple AWS services.

Note: Ingestion of AWS metrics into Coralogix using CloudWatch API offers a simpler and more cost-effective method of metrics retrieval. Refer to AWS Metrics CloudWatch for integration instructions.

Setup Options

• Automated Integration Package (Recommended). Streamline the ingesting and analyzing metrics from your AWS resources using our automated integration package setup below.

• Manual Integration. Alternatively, use our manual integration below.

• Terraform. Install and manage the Firehose Metrics integration with AWS services as modules in your infrastructure code.

• CloudFormation. Install our AWS Firehose Metrics with CloudFormation Template and incorporate the necessary configurations and settings via CloudFormation to automate your Firehose metrics collection setup and management.

Automated Integration Package

Streamline your setup process using our automated integration package CloudWatch Metrics via Firehose. This package lets you preconfigure and deploy a template, replicating the manual setup.

Once completed, select from various out-of-the-box extension packages: AWS EBS, AWS EC2, AWS ElastiCache, or AWS RDS.

Each tailored extension unlocks a set of predefined items – such as alerts, parsing rules, dashboards, saved views, and actions – allowing you to jumpstart Coralogix monitoring of your external-facing resources.

Configuration

STEP 1. From your Coralogix toolbar, navigate to Data Flow > Integrations.

STEP 2. In the Integrations section, select CloudWatch Metrics via Firehose.

STEP 3. Click ADD NEW.

STEP 4. Input your integration details.

• Integration Name. Enter a name for your integration and either enter your Send-Your-Data API key or click CREATE A NEW KEY to create a new API key for the integration.

• Application Name. Enter an application name. The default name is AWS.

• Subsystem Name. Enter a subsystem name. The default name is Firehose.

• CloudWatch Metric Namespaces. Select the namespaces you want to bring from the dropdown menu into the Coralogix platform.

• Enable Additional Statistics. Enabling additional statistics is recommended to receive detailed percentile data on key AWS metrics. However, enabling it may incur additional costs from AWS. The configuration used for the additional statistics is as follows:

[ { "AdditionalStatistics": ["p50", "p75", "p95", "p99"], "IncludeMetrics": [ {"Namespace": "AWS/EBS", "MetricName": "VolumeTotalReadTime"}, {"Namespace": "AWS/EBS", "MetricName": "VolumeTotalWriteTime"}, {"Namespace": "AWS/ELB", "MetricName": "Latency"}, {"Namespace": "AWS/ELB", "MetricName": "Duration"}, {"Namespace": "AWS/Lambda", "MetricName": "PostRuntimeExtensionsDuration"}, {"Namespace": "AWS/S3", "MetricName": "FirstByteLatency"}, {"Namespace": "AWS/S3", "MetricName": "TotalRequestLatency"} ] } ]

• AWS Region. Select your AWS region from the dropdown menu.

Optional [Recommended]: Enable Metrics Tags Processors by using the mentioned Transformation Lambda to enhance metrics by adding AWS resource tags using the abovementioned settings.

STEP 6. Click NEXT.

STEP 7. Review the instructions for your integration and click CREATE CLOUDFORMATION.

STEP 8. You will be rerouted to the AWS website. Verify that all of the auto pre-populated values are correct. Click Create Stack.

STEP 9. Go back to the Coralogix application and click COMPLETE to close the module. Revert to the integration page.

STEP 10. [Optional] Deploy the extension package of your choice to complement your integration needs.

• AWS EBS

• AWS EC2

• AWS ElastiCache

• AWS RDS

STEP 11. View the metrics in your Coralogix dashboard using Custom Dashboards [recommended] or hosted Grafana View.

Manual Setup

Prerequisites

• Metrics bucket configured in your Coralogix dashboard. Data Flow > Setup Archive

Configuration

STEP 1. Go to the Amazon Data Firehose console and choose ‘Create delivery stream’.

STEP 2. Under ‘Choose source and destination’.

• Source: Direct PUT

• Destination: Coralogix

• Delivery stream name: Fill in the desired stream name.

STEP 3. Scroll down to ‘Destination settings’.

• HTTP endpoint URL: Choose a Firehose HTTP endpoint URL based on your Coralogix domain and region.

STEP 4. Scroll down to ‘Parameters’:

• Private key: Enter your Coralogix Send Your Data – API Key.

• Content encoding: Select GZIP.

• Retry duration: Choose 300 seconds.

By default, your delivery stream arn and name will be used as ‘applicationName’ and ‘subsystemName’.

Add a new parameter with the desired value to override the associated 'applicationName' or 'subsystemName'.

• Key: 'applicationName' , value - 'new-app-name'

• Key: 'subsystemName' , value - 'new-subsystem-name'

The source of the data in Firehose determines the ‘integrationType’ parameter value:

• Key: ‘integrationType’ , value: ‘CloudWatch_Metrics_OpenTelemetry070_WithAggregations′

STEP 5. Scroll down to ‘Backup settings’:

• Source record backup in Amazon S3: We suggest selecting ‘Failed data only’.

• S3 backup bucket: Choose an existing bucket or create a new one.

• Buffer hints, compression, encryption: Leave these fields as is.

STEP 6. Review your settings and choose ‘Create delivery stream’.

Metrics subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.

Data Source Configuration

Cloudwatch Metrics

To start sending your metrics to Coralogix, you first need to create a metric stream.

STEP 1. Go to the Cloudwatch console and choose ‘Streams’ under the ‘Metrics’ side menu.

STEP 2. Click on ‘Create metric stream’.

STEP 3. Under ‘Metrics to be streamed’:

• Choose what metrics to send

STEP 4. Scroll down to ‘Configuration’:

• For ‘Select configuration option’ choose ‘Select an existing Firehose owned by your account’

• For ‘Select your Amazon Data Firehose stream’ choose the delivery stream created above

STEP 5. Scroll down to ‘Change output format’

• Make sure that ‘OpenTelemetry 0.7’ is selected

STEP 6. Scroll down to ‘Custom metric stream name’ and pick a name for the metrics stream.

STEP 7. Scroll down and click on ‘Create metric stream’.

After a few minutes, the metrics will start streaming to Coralogix, and you will see them on the Grafana dashboard.

Transformation Lambda

[Optional] CloudWatch Metric Streams Lambda transformation function can be used as a Amazon Data Firehose transformation function, to enrich the metrics from CloudWatch Metric Streams with AWS resource tags.

This installation is optional, and you can install the transformation Lambda if you’d like to take advantage of having AWS resource tags as labels in your metrics data. Find out more here.

Take the following steps to install the transformation Lambda.

STEP 1. Create a new AWS Lambda function in your designated region with the following parameters:

• Runtime: Custom runtime on Amazon Linux 2

• Handler: bootstrap

• Architecture: arm64

STEP 2. Download the bootstrap ZIP file from the releases in the repository. Unless instructed otherwise, use the latest release. Upload the bootstrap.zip as the code source for Lambda.

STEP 3. Make sure to set the memory. We recommend starting with 512 MB and, depending on the number of metrics you export and the speed of Lambda processing, see if you need to increase it.

STEP 4. Adjust the role of the Lambda function as described in the necessary permissions.

STEP 5. Optionally, add environment variables to configure the Lambda, as described in the configuration.

STEP 6. The Lambda function is ready for use in Amazon Data Firehose Data Transformation. Please note the ARN and provide it in the relevant section of the Amazon Data Firehose configuration.

To prevent delay in the delivery of your data and depending on the size of your setup, we recommend adjusting your Lambda buffer hint and Amazon Data Firehose buffer size configuration accordingly. For the most optimal experience, we recommend setting the Lambda buffer hint  0.2 MB and the Amazon Data Firehose buffer size to 1 MB. Note that this may cause more frequent Lambda runs, resulting in higher costs.

You can check the staleness of your data in your Amazon Data Firehose delivery stream in the Monitoring tab by looking at the ‘Delivery to HTTP endpoint data freshness’. Seeing the staleness value grow may indicate that the Lambda function runs too slowly. In such a case, you may increase the Lambda’s memory and set the buffering configuration as described above.

Terraform - Module Setup

Using Coralogix Terraform modules, you can easily install and manage Coralogix integrations with AWS services as modules in your infrastructure code. Our open-source modules are available on our GitHub and in the Terraform Registry. Visit our full Amazon Data Firehose Terraform Module documentation for more information.

Metrics

For metrics, install Amazon Data Firehose by also adding this declaration to your Terraform project:

module "cloudwatch_firehose_coralogix_metrics" {
  source           = "coralogix/aws/coralogix//modules/firehose-metrics"
  private_key      = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  firehose_stream  = "coralogix-firehose-metrics"
  coralogix_region = "Europe"
}

Cross-Account Metric Streams

Support for cross-account Metric Streams in Amazon CloudWatch was recently announced. CloudWatch Metric Stream (Monitoring Accounts) can pull metrics from other AWS accounts (Source Accounts), centralizing your Firehose and Metric Stream deployment.

After following the CloudWatch cross-account setup guide on AWS, in your monitoring account’s console. Select CloudWatch > Metric Streams, select the metric stream, and click the Edit button.

The following toggle switch “Include source account metrics” will appear, enabling it to retrieve source account metrics.

Note: The Transformation Lambda can only access AWS resource tags from the monitoring account; thus, only monitoring account metrics data will be enriched.

Amazon CloudWatch Extensions

Coralogix offers a variety of out-of-the-box data extension packages: AWS EBS, AWS EC2, AWS ElastiCache, or AWS RDS.

Each tailored extension unlocks a set of predefined items – such as alerts, parsing rules, dashboards, saved views, and actions – allowing you to jumpstart Coralogix monitoring of your external-facing resources.

Destination Errors

Review these common destination errors presented by Firehose and their possible solutions

Limitations

CloudWatch Metric Streams does not send metrics that have a timestamp older than 2 hours. This means that some CloudWatch metrics that are calculated at the end of a day and reported with the beginning timestamp of the same day would not be streamed. This includes S3 daily storage metrics and some billing metrics.

Should you need these metrics, we recommend using Cloudwatch Exporter using Prometheus alongside our new CloudWatch integration designed to retrieve those metrics. For updated information, contact Coralogix Support.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Contact us via our in-app chat or by emailing [email protected].

Previous Send Logs using Amazon Data Firehose

Next Forward AWS Logs via Lambda Shipper

Theme

Light

light

dark