Fluentd

Coralogix provides seamless integration with Fluentd so you can send your logs from anywhere and parse them according to your needs.

Prerequisites

Fluentd installed

Parameters

Every configuration requires that you supply your Coralogix Send-Your-Data API key (Api_Key) and the Coralogix REST API Singles endpoint associated with your Coralogix domain.

Configuration

The following section provides a number of configuration examples.

Generic Configuration

The example below collects all log files from the /var/log/ path, using the generic HTTP output plugin. We recommend this plugin due to its adjustability and exposed metrics. Other collection options can be found here.

<system>
    log_level info
</system>

<source>
    @type tail
    @id tail_var_logs
    @label @CORALOGIX
    path /var/log/*.log
    pos_file /var/log/fluent/all.pos
        path_key path
    tag all
    read_from_head true
    <parse>
        @type none
    </parse>
</source>

<label @CORALOGIX>
    <filter **>
    @type record_transformer
    @log_level warn
    enable_ruby true
    auto_typecast true
    renew_record true
    <record>
        applicationName "Example_App"
        subsystemName "Example_Subsystem"
        text ${record.to_json}
    </record>
    </filter>

<match **>
    @type http
    @id http_to_coralogix
    endpoint "https://ingress.<span class="domain-value"></span>/logs/v1/singles"
    headers {"authorization":"Bearer <Api_Key>"}
    retryable_response_codes 503
    error_response_as_unrecoverable false
    <buffer>
        @type memory
        chunk_limit_size 10MB
        compress gzip
        flush_interval 1s
        retry_max_times 5
        retry_type periodic
        retry_wait 2
    </buffer>
    <secondary>
        #If any messages fail to send they will be send to STDOUT for debug.
        @type stdout
    </secondary>
</match>
</label>

Configuration for Multiple Sources

The example below provides a Fluentd configuration that has several log file sources and a unique subsystem name for each source. It uses the "tag" option to specify a unique value in each source and a "tag" variable as subsystem. The result is several different tags depending on the source of the log.

<system>
  log_level info
</system>

<source>
  @type tail
  @id tail_path_1
  @label @CORALOGIX
  path /<path number 1>/*/*.log
  pos_file /<path number 1>/all.pos
        path_key path
  tag path_number_1
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<source>
  @type tail
  @id tail_path_2
  @label @CORALOGIX
  path /<path number 2>/*/*.log
  pos_file /<path number 2>/all.pos
        path_key path
  tag path_number_2
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<label @CORALOGIX>
  <filter **>
  @type record_transformer
  @log_level warn
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    applicationName "<Application Name Here>"
    subsystemName ${tag}
    text ${record.to_json}
  </record>
  </filter>

<match **>
  @type http
  @id http_to_coralogix
  endpoint "https://ingress.<span class="domain-value"></span>/logs/v1/singles"
  headers {"authorization":"Bearer <Api_Key>"}
  retryable_response_codes 503
  error_response_as_unrecoverable false
  <buffer>
    @type memory
    chunk_limit_size 10MB
    compress gzip
    flush_interval 1s
    retry_max_times 5
    retry_type periodic
    retry_wait 2
  </buffer>
  <secondary>
    #If any messages fail to send they will be send to STDOUT for debug.
    @type stdout
  </secondary>
</match>
</label>

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

Beats: Filebeat

Next GCP - Getting Started