Overview

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that allows the use of Apache Kafka for streaming data processing. This guide details how to deploy an AWS MSK and prepare it to receive parsed and enriched data from Coralogix. It’s a companion guide to the Forwarders documentation, which explains how to set up and configure Coralogix to send your telemetry data to AWS MSK.

Deployment

Use one of the following deployment methods:

• Terraform

• Manual

Terraform

STEP 1. Run the following file to deploy an AWS MSK. If needed, refer to this readme file for more deployment options.

Note: Module creation might take between 40 and 90 minutes.

module "msk-data-stream" {
  source = "coralogix/aws/coralogix//modules/msk-data-stream"

  cluster_name = <your cluster name>
}

STEP 2. After the module is created, copy the public endpoints of the cluster brokers. Use them in the AWS MSK URLs field of the Forwarder Setup dialog box.

Manual

Prerequisites

• VPC

• 3 public subnets

Configuration

STEP 1. In the Search bar of the AWS Management Console, search for MSK, then click Create cluster.

STEP 2. Enter the following cluster parameters and click Next.

Note: you can increase the storage according to your requirements.

STEP 3. In the Networking section, select an existing VPC.

STEP 4. In the Zones section, select your zones and choose the public subnets.

STEP 5. In the Security groups (Amazon EC2 section), click Browse, select your security group, and click Choose.

STEP 6. Make sure that your security group allows Coralogix to access the MSK.

STEP 7. In the Security settings section, enable the IAM role-based authentication option and click Next.

STEP 8. In the Monitoring section, select Basic monitoring, then click Next.

Note: Optionally, you can enable the Deliver to Amazon CloudWatch Logs, which is useful for troubleshooting. If you do, choose your log group to use it.

STEP 9. Review your configuration. If it’s OK, click Create cluster. Wait 15-45 minutes for your cluster to activate.

STEP 10. After the cluster becomes active, select Edit public access and turn it on, then save the changes. The cluster will be updated. This might take up to 40 minutes.

Update cluster policy

After the cluster has been updated and became active, copy and paste this policy.

  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<coralogix_role_arn>"
        ]
      },
      "Action": "kafka-cluster:Connect",
      "Resource": "< your_msk_cluster_arn >"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "< coralogix_role_arn >"
        ]
      },
      "Action": [
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": "arn:aws:kafka:< aws_region >:< aws_account_id >:topic/< your_cluster_name >/*"
    }
  ]

Replace the following parameters with your values:

• your_aws_account_id - Your AWS account ID

• aws_region - The AWS region that you deployed the MSK in

• your_cluster_name - The cluster name

• your_msk_cluster_arn - Your MSK cluster ARN

• aws_account_id - Your AWS account ID

• coralogix_role_arn - Copy the role from the following map:

  • EU1 - arn:aws:iam::625240141681:role/coralogix-archive-eu1

  • EU2 - arn:aws:iam::625240141681:role/coralogix-archive-eu2

  • US1 - arn:aws:iam::625240141681:role/coralogix-archive-us1

  • US2 - arn:aws:iam::739076534691:role/coralogix-archive-us2

  • AP1 - arn:aws:iam::625240141681:role/coralogix-archive-ap1

  • AP2 - arn:aws:iam::625240141681:role/coralogix-archive-ap2

Additional settings

• To create a new topic, connect to a machine that is in the same VPC as the MSK. Alternatively, connect to a machine with the AWS configuration for the account with the MSK. Once you are in the machine, run the following commands:

  • sudo yum install -y java-1.8*

  • wget <https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-3.5.1.tgz> && tar -xzvf kafka_2.13-3.5.1.tgz && rm kafka_2.13-3.5.1.tgz

  • kafka_2.13-3.5.1/bin/kafka-topics.sh --create --bootstrap-server <public brokerst list > --topic <Topic-Name> --partitions 3 --replication-factor 3

• To restrict Coralogix access only to this topic, edit the cluster policy by changing the topic resource (Resource": "arn:aws:kafka:<region>:<Account-ID>:topic*") to "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:topic/<CLUSTER_NAME>/<TOPIC_NAME>"

• Extract public endpoint for brokers.

Additional resources

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email to [email protected].

Theme

Light

light

dark