SyslogNG

Determining syslog type

Coralogix supports both rsyslog and syslog-ng.

Open your terminal window and type:

ls -d /etc/*syslog*

If you see rsyslog.d, you are using rsyslog. If you see syslog-ng, you are using syslog-ng.

If you don't see any of these options then please install rsyslog or syslog-ng. Most linux distributions already have one of these syslog packages so you should refer to documentation of your linux distribution for installation guidelines.

syslog-ng configuration

Open your terminal window and download the coralogix syslog-ng configuration file coralogix.client.syslog-ng.conf (right click link + save) and save it under /etc/syslog-ng/conf.d/ folder.

cd /etc/syslog-ng/conf.d && wget https://coralogixstorage.blob.core.windows.net/syslog-configs/syslog-configs/coralogix.client.syslog-ng.conf

Open the coralogix.client.syslog-ng.conf file with your favorite editor:

vi /etc/syslog-ng/conf.d/coralogix.client.syslog-ng.conf

Template configuration

Navigate to the template section and update the CoralogixSyslogFormat parameter with your specific values. You should change only the values in BOLD:

#*****************************************************************#
#                        TEMPLATE SECTION                         #
#*****************************************************************#

template CoralogixSyslogFormat { template("crx=1 crxversion=1 crxtype=syslog crxcompid=[YOUR COMPANY 
ID] crxpkey=[YOUR COMPANY KEY] crxapp=[YOUR APPLICATION NAME] crxsubsys=[YOUR APPLICATION SUBSYSTEM 
NAME] crxhostname=${HOST} crxtag=${TAGS} <CRX.TIME_STAMP=${S_UNIXTIME}${R_MSEC}> <CRX.MSEC=${R_MSEC}> 
<CRX.PRI=${FACILITY}.${LEVEL}> <CRX.SEVERITY_REWRITE=''> <CRX.CATEGORY_REWRITE=''> 
<CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=${PROGRAM}> <CRX.MSG=$MSG>\n"); };

[YOUR COMPANY ID]: A unique ID which represents your company, this ID will be sent to your mail once you register to Coralogix.

[YOUR COMPANY KEY]: You can locate your company key in the Coralogix dashboard. Please navigate to Settings->SEND YOUR LOGS

[YOUR APPLICATION NAME]: The Application name parameter allows you to split between the different sources of your data, whether it’s different environments or complete different applications.

[YOUR APPLICATION SUBSYSTEM NAME]: Your application probably has multiple subsystems, for example: Backend servers, Middleware, Frontend servers etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

If you have several applications or subsystem components writing to the same syslog then you should create this template for each and one of them and give each a unique name. For instance, if you have an application with the name myapp and under that application you have 2 subsystems: mydal and and myclient (2 different processes running on the same host). An appropriate template configuration would be:

template CoralogixSyslogFormat1 { template("crx=1 crxversion=1 crxtype=syslog crxcompid=1 
crxpkey=11111111-1111-1111-1111-1111111111 crxapp=myapp crxsubsys=mydal crxhostname=${HOST} 
crxtag=${TAGS} <CRX.TIME_STAMP=${S_UNIXTIME}${R_MSEC}> <CRX.MSEC=${R_MSEC}> 
<CRX.PRI=${FACILITY}.${LEVEL}> <CRX.SEVERITY_REWRITE=''> <CRX.CATEGORY_REWRITE=''> 
<CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=${PROGRAM}> <CRX.MSG=$MSG>\n"); };

template CoralogixSyslogFormat2 { template("crx=1 crxversion=1 crxtype=syslog crxcompid=1 
crxpkey=11111111-1111-1111-1111-1111111111 crxapp=myapp crxsubsys=myclient crxhostname=${HOST} 
crxtag=${TAGS} <CRX.TIME_STAMP=${S_UNIXTIME}${R_MSEC}> <CRX.MSEC=${R_MSEC}> 
<CRX.PRI=${FACILITY}.${LEVEL}> <CRX.SEVERITY_REWRITE=''> <CRX.CATEGORY_REWRITE=''> 
<CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=${PROGRAM}> <CRX.MSG=$MSG>\n"); };

Filter configuration

Navigate to the filter section of the file:

#*****************************************************************#
#                         FILTER SECTION                          #
#*****************************************************************#

#Filter messages and send only the relevant one
#For more information and other filter options please refer to: 
#https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-filters.html
filter f_coralogix {
    #Use a regexp on program name:
    #program(^myApp$); #This will filter messages and send only the one with program name equal to: myApp

    #Filter by facility
    #facility(user); #This will filter messages and send only the one with facility equal to: user

    #Filter by message
    message("(.*?)") #Match any message
};

You should configure syslog to send logs only from your application rather then the entire messages coming from your linux OS. Using syslog-ng filters you are basically forwarding only those messages that successfully passed your filter. You can define many different filters. For example:

To filter only application with the name myapp:

filter f_coralogix {program(^myApp$);};

To filter only applications writing to facility user:

filter f_coralogix {facility(user);};

If you have several applications or subsystem components writing to the same syslog then you should create a new filter for each one and give each a unique name.

You can also filter by other parameters and you can use regular expressions as well. For more information on filtering please refer to: syslog-ng filters

Destination configuration

Navigate to destination section.

#*****************************************************************#
#                       DESTINATION SECTION                       #
#*****************************************************************#
#Configure Coralogix UDP destingation
destination d_coralogix { network("syslogserver.coralogix.com" transport("udp") port(514) template(CoralogixSyslogFormat) log-fifo-size(1000)); };

#Configure Coralogix TCP destingation
#destination d_coralogix { network("syslogserver.coralogix.com" transport("tcp") port(1514) template(CoralogixSyslogFormat) log-fifo-size(1000)); };

#Optionally you can configure logs to be redirected to your local file as well. This is useful if you want to see the exact message that is being sent to the Coralogix server.
#destination d_local { file("/var/log/messages" template(CoralogixSyslogFormat)); };

#Send the logs from the source to the Coralogix server with your filter
log { source(s_coralogix); filter(f_coralogix); destination(d_coralogix); flags(flow-control);};

#If you configured a local destination then you should uncomment this line to send messages to your local file.
#log { source(s_coralogix); filter(f_coralogix); destination(d_local); };

You can configure syslog-ng to send logs via TCP or UDP protocol. By default the configuration file is configured for UDP. If you prefer TCP then please comment the UDP line and uncomment the TCP one. Coralogix is using port 514 for UDP and 1514 for TCP

In addition you can redirect your syslog messages to your local file. This is useful if you want to see the exact data that is being sent to Coralogix. This option is great for debugging. If you are having troubles sending your syslog data to Coralogix, then you should first check logs are written locally. Another good example can be in case you want to filter your syslog data based on your application name but you are not sure the exact name of the process. If you want to use this option then just uncomment this line:

destination d_local { file("/var/log/messages" template(CoralogixSyslogFormat)); };

Here is an example for the log output of an application myapp sending log: Hello World!:

tail -f /var/log/messages

Nov 10 21:10:06 127.0.0.1/127.0.0.1 crx=1 crxversion=1 crxtype=syslog crxcompid=1
crxpkey=11111111-1111-1111-1111-1111111111 crxapp=myapp crxsubsys=mydal crxhostname=hostname1
crxtag='' <CRX.TIME_STAMP=1479218614752> <CRX.MSEC=752> <CRX.PRI=daemon.err> <CRX.CATEGORY_REWRITE=''>
<CRX.SEVERITY_REWRITE=''> <CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=my-app1> <CRX.MSG=Hello World!>

crxapp=myapp - This is the name of the application as you defined it in the template.

<CRX.PROGRAM_NAME=my-app1> - This is the process/program name that sent the log line to syslog.

If you defined several templates for each program/process name then instead of using one generic redirect rule, you should use a filter based redirect. For instance, you have two different sub systems (mydal and myclient) part of the same application (myapp) writing to the same system. Here is an optional configuration for this scenario:

template CoralogixSyslogFormat1 { template("crx=1 crxversion=1 crxtype=syslog crxcompid=1 
crxpkey=11111111-1111-1111-1111-1111111111 crxapp=myapp crxsubsys=mydal crxhostname=${HOST} 
crxtag=${TAGS} <CRX.TIME_STAMP=${S_UNIXTIME}${R_MSEC}> <CRX.MSEC=${R_MSEC}> 
<CRX.PRI=${FACILITY}.${LEVEL}> <CRX.SEVERITY_REWRITE=''> <CRX.CATEGORY_REWRITE=''> 
<CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=${PROGRAM}> <CRX.MSG=$MSG>\n"); };

template CoralogixSyslogFormat2 { template("crx=1 crxversion=1 crxtype=syslog crxcompid=1 
crxpkey=11111111-1111-1111-1111-1111111111 crxapp=myapp crxsubsys=myclient crxhostname=${HOST} 
crxtag=${TAGS} <CRX.TIME_STAMP=${S_UNIXTIME}${R_MSEC}> <CRX.MSEC=${R_MSEC}> 
<CRX.PRI=${FACILITY}.${LEVEL}> <CRX.SEVERITY_REWRITE=''> <CRX.CATEGORY_REWRITE=''> 
<CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=${PROGRAM}> <CRX.MSG=$MSG>\n"); };

filter f_coralogix_dal {program(^mydal$);};
filter f_coralogix_client {program(^myclient$);};

destination d_coralogix_dal { network("syslogserver.coralogix.com" transport("udp") port(514) 
template(CoralogixSyslogFormat1) log-fifo-size(1000)); };
destination d_coralogix_client { network("syslogserver.coralogix.com" transport("udp") port(514) 
template(CoralogixSyslogFormat2) log-fifo-size(1000)); };

log { source(s_coralogix); filter(f_coralogix_dal); destination(d_coralogix_dal); flags(flow-
control);};
log { source(s_coralogix); filter(f_coralogix_client); destination(d_coralogix_client); flags(flow-
control);};

Save the file and restart syslog-ng. The command to restart syslog-ng daemon can vary from one Linux distribution to another but in most cases this would be:

sudo service syslog-ng restart

Testing configuration

At this point you should be ready to test your configuration. To send a test message you can use the linux logger command:

logger -p info Hello World!

This should send a message "Hello World!" with severity info. If you enabled the option to redirect your syslog messages to a local file then you should see this message with the command:

tail -f /var/log/messages

If you don't see the message then please check your configuration.

Next, navigate to 'Log Query' menu in the Coralogix dashboard and press the Go button to search for the last 15 minutes logs. If you see your log then CONGRATULATIONS! You are now connected to Coralogix.

Still not viewing your logs ?, book your implementation session, and we'll make sure your logs are right where they should be.

Docker syslog configuration

Docker provides several log drivers that can redirect console output logs to a log server. For a complete list of log drivers please refer to: Docker log drivers

This section describes how to work with Docker syslog driver and redirect your messages to Coralogix server.

Make sure that the host running your Docker container has a syslog daemon up and running.

ps aux | grep syslog

Configure your syslog to forward messages to Coralogix server. Please see section 2 for rsyslog or section 3 for syslog-ng.
Run your docker with the --log-driveroption, for instance:

docker run -d --log-driver\=syslog ubuntu /bin/sh -c "while true; do echo hello world; sleep 1; done"