Connect S3 Archive

This tutorial demonstrates configuring an S3 bucket to send your telemetry data to Coralogix.

Use your archive buckets to store all of your data as long as you need, on your own S3 bucket, for infinite retention. Your logs & traces archive bucket can be stored in CX data format. Metrics archive bucket stores Prometheus index blocks.

Users have three configuration options:

• CloudFormation

• Terraform module

• Manual

Configuration with CloudFormation

STEP 1. Create a new stack.

• In the Amazon S3 URL field, input the following URL: https://cgx-cloudformation-templates.s3.eu-central-1.amazonaws.com/coralogix-s3-archive/template.yaml. This will create the required bucket with the necessary policies.

• Run the stack in the same region in which the bucket is created.

• Make sure to match your bucket region with the region associated with your Coralogix account.

STEP 2. Specify your stack details.

• Input a unique LogsBucketName and/or MetricsBucketName.

• If your bucket needs to be encrypted and you are using KMS, ensure that the KMS is in the same region as your bucket.

• Parameters that you don’t need may be left empty.

• Follow AWS bucket naming rules.

STEP 3. Run the template to create the S3 archives.

Terraform

STEP 1. Access the S3-archive module in the Coralogix Terraform Registry.

STEP 2. Create the Terraform file with the following variables.

Notes:

• Bucket names must follow AWS AWS bucket naming rules.

• View more configuration options here.

STEP 3. [Optional] Add KMS keys to the buckets with logs_kms_arn and metrics_kms_arn variables.

STEP 4. Create the archive buckets by saving the file and running the following commands:

terraform init

terraform apply

Manual Configuration

STEP 1. Create an S3 bucket for configuration with default parameters.

STEP 2. Search S3 in your AWS search bar and select this service.

STEP 3. Locate and select the bucket of choice for storing the archive.

STEP 4. Navigate to the Permissions tab. Edit the Bucket policy.

STEP 5. Paste the following code and update the name of your bucket. The following provides both a logs and metrics archive bucket configuration snippet. Note that the configuration snippet for metrics contains an additional delete permission.

Notes:

• The configuration snippet for metrics contains an additional delete permission.

• For all configured snippets, you must input the ACCOUNT_ID associated with your Coralogix domain (e.g. "coralogix.us") and region (e.g. us1).

Logs & Traces Archive Bucket Configuration Snippet

{
    "Version": "2012-10-17",
    "Id": "MyPolicyID",
    "Statement": [
        {
            "Sid": "MyStatementSid",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/coralogix-archive-<REGION>"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObjectTagging",
                "s3:DeleteObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>",
                "arn:aws:s3:::<bucket name>/*"
            ]
        }
    ]
}

Metrics Archive Bucket Configuration Snippet

{
    "Version": "2012-10-17",
    "Id": "MyPolicyID",
    "Statement": [
        {
            "Sid": "MyStatementSid",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<Account_ID>:root"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObjectTagging",
                "s3:DeleteObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>",
                "arn:aws:s3:::<bucket name>/*"
            ]
        }
    ]
}

STEP 6. Click Save changes.

STEP 7. Copy and save the name of the archive for your Coralogix platform setup.

KMS Encrypted Bucket (Optional)

If your bucket needs to be encrypted and you are using KMS, you will need to grant Coralogix permission to use the specific key to encrypt and decrypt the data when we store it on your bucket. Only those keys that are ‘Customer Managed’ may have their policy modified.

STEP 1. Access KMS and choose the key used for encryption. Note: The key must be in the same region as your archive bucket.

STEP 2. Edit the Key Policy and add the highlighted section below to your KMS key policy or copy the code to give your root user full access.

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable Coralogix Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/coralogix-archive-<REGION>"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Enable root Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<your-user>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
     ]
}

STEP 3. Add the key to the S3 bucket. Navigate go to your archive bucket. Under Properties, edit the Default encryption.

STEP 4. Select Server-side encryption with AWS Key Management Service keys (SSE-KMS) and insert your KMS ARN. Enable the Bucket Key. Click Save changes.

Notes:

• For key policy, an asterisk “*” on the resource field refers to the specific KMS key and not all resources.

Coralogix Platform Setup

STEP 1. Select the Data Flow tab and click on Setup Archive.

STEP 2. Choose from two bucket options: one for logs & traces (CX data) and one for metrics. You cannot use the same bucket for metrics and logs together.

STEP 3. Click Save. 

Notes:

• This will activate your Archive Storage settings. Find out more here.

STEP 4. Query your Metrics S3 bucket through our hosted Grafana metric index or use our plugin and input metrics_index.

Avoiding Excess S3 Costs

Avoid excess S3 costs when using Coralogix direct S3 queries.

Overview of Amazon S3 Infrequent Access Storage Class

Amazon S3 Infrequent Access storage class is ideally suited for data that isn’t accessed frequently but requires quick accessibility when needed. This storage class offers a cost-effective solution for storing such data. Coralogix customers must understand the specific cost implications of accessing data within this tier.

Coralogix Integration & Impact on Cost

For applications utilizing Coralogix in conjunction with Amazon S3's Infrequent Access storage, it’s critical to be aware that accessing data stored in this tier might lead to additional costs, especially for querying the data from the Coralogix UI. This is due to the unique pricing structure Amazon S3 applies to data in the infrequent access category.

Recommended Action

To ensure optimal data storage management and avoid unexpected costs, we advise all of our customers to review AWS S3 pricing and familiarize themselves with the details of AWS S3 pricing, especially the sections on the Infrequent Access storage class. Using any tier other than S3 Standard may incur additional AWS costs.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Contact us via our in-app chat or by emailing [email protected].

Previous Forwarders

Next Archive Retention Policy

Theme

Light

light

dark