Kubernetes with Fluentd (Without Helm)

Fluentd is a versatile data shipper with numerous available plugins and functionalities, playing a pivotal role as a logs shipper to our platform. Below are instructions on how to set up the Fluentd shipper along with the http output plugin to transmit logs to the Coralogix platform.

Prerequisites

Before you begin, make sure you have:

Enabled RBAC authorization mode in your Kubernetes Cluster
Your Coralogix Send-Your-Data API key

Installation

To specify essential environment variables, create a configmap as shown below:

---
## fluentd-env-cm.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: fluentd
    app.kubernetes.io/instance: fluentd-http
  name: fluentd-env
data:
  ENDPOINT: ingress.<span class="domain-value"></span>
  LOG_LEVEL: error

Notes:

The configmap name is crucial and utilized by the daemonSet.
Ensure to change 'ENDPOINT' according to your logs endpoint from the table provided.

Apply the configmap:

kubectl apply -f fluentd-env-cm.yaml -n monitoring

Next, apply the manifest files in this directory:

kubectl apply -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-cm.yaml -n monitoring
kubectl apply -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-rbac.yaml -n monitoring
kubectl apply -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-svc.yaml -n monitoring
kubectl apply -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-ds.yaml -n monitoring

You should see the following output:

configmap/fluentd-prometheus-conf created
configmap/fluentd-systemd-conf created
configmap/fluentd-config created
configmap/fluentd-main created
daemonset.apps/fluentd-http created
serviceaccount/fluentd-http created
clusterrole.rbac.authorization.k8s.io/fluentd-http created
clusterrolebinding.rbac.authorization.k8s.io/fluentd-http created
servicemonitor.monitoring.coreos.com/fluentd-http created
service/fluentd-http created

If you have the prometheus-operator installed, you can install this service monitor resource:

kubectl apply -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-svc-monitor.yaml -n monitoring

Modifying applicationName and subsystemName

By default, we utilize the field kubernetes.namespace_name as the applicationName and kubernetes.container_name as the subsystemName.

Dynamic

To modify these values and use another field as the value of applicationName and subsystemName, modify the fluentd-config configmap and specifically the coralogix.conf key.

Static

To modify these values and use a hard-coded value as the value of applicationName and subsystemName, modify the fluentd-config configmap and specifically the coralogix.conf key.

Removal

To remove all resources created with manifest files, use these commands:

kubectl delete -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-cm.yaml -n monitoring
kubectl delete -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-rbac.yaml -n monitoring
kubectl delete -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-svc.yaml -n monitoring
kubectl delete -f https://raw.githubusercontent.com/coralogix/telemetry-shippers/master/logs/fluentd/k8s-manifest/fluentd-ds.yaml -n monitoring
kubectl delete -f fluentd-env-cm.yaml -n monitoring

The output should be:

configmap "fluentd-prometheus-conf" deleted
configmap "fluentd-systemd-conf" deleted
configmap "fluentd-config" deleted
configmap "fluentd-main" deleted
daemonset.apps "fluentd-http" deleted
serviceaccount "fluentd-http" deleted
clusterrole.rbac.authorization.k8s.io "fluentd-http" deleted
clusterrolebinding.rbac.authorization.k8s.io "fluentd-http" deleted
servicemonitor.monitoring.coreos.com "fluentd-http" deleted
service "fluentd-http" deleted
configmap "fluentd-env" deleted

Coralogix endpoints

Region	Logs Endpoint
EU	`ingress.coralogix.com`
EU2	`ingress.eu2.coralogix.com`
US	`ingress.coralogix.us`
SG	`ingress.coralogixsg.com`
IN	`ingress.coralogix.in`

Deploy to a different namespace

If you need to deploy the Fluentd integration to a namespace other than "monitoring", change the fluentd-rbac.yaml file ClusterRoleBinding namespace accordingly.

Disable Systemd Logs

To disable the systemd logs, remove the fluentd-systemd-conf configmap:

kubectl delete cm fluentd-systemd-conf

Dashboard

Under the dashboard directory, you'll find a Fluentd Grafana dashboard provided by Coralogix. To import the dashboard into Grafana, copy the JSON file content, navigate to Grafana, click on the Create tab, then import, and paste the copied JSON file.