AWS Kinesis with Logstash

Coralogix provides integration to connect Logstash to AWS Kinesis , so you can send your logs from anywhere into Coralogix.

Prerequisites

Logstash installed

Usage

You must provide the following four variables when creating a Coralogix logger instance.

Private Key – Your Send Your Data - API Key is a unique ID that represents your company.

Application Name – The name of your environment, for example, a company named “SuperData” would probably insert the “SuperData” string parameter or if they want to debug their test environment they might insert the “SuperData– Test”.

SubSystem Name – Your application probably has multiple components, for example: Backend servers, Middleware, Frontend servers etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

Region - The AWS region for Kinesis.

Installation

logstash-plugin install logstash-input-kinesis
logstash-plugin install logstash-output-coralogix

If you are not sure where the logstash-plugin is located, you can check this here.

Configuration

Open your Logstash configuration file and add AWS Kinesis input and Coralogix.

input {
  kinesis {
    kinesis_stream_name => "XXXXXXXX"
    region => "XX-XXXX-X"
    codec => json
  }
}

output {
    coralogix {
        config_params => {
            "PRIVATE_KEY" => "YOUR_PRIVATE_KEY"
            "APP_NAME" => "APP_NAME"
            "SUB_SYSTEM" => "SUB_NAME"
        }
        log_key_name => "message"
        timestamp_key_name => "@timestamp"
        is_json => true
    }
}

Input

kinesis_stream_name is mandatory.
region is optional (Default value is "us-east-1").

More information about how to setup Logstash Input Kinesis plugin: logstash-input-kinesis.

Output

Watch more information about our logstash-output-coralogix plugin.

AWS VPC Flow Logs

Next AWS Load Balancer