Elasticsearch

Today, one of the easiest ways to ship logs from a Kubernetes cluster is by using Fluent Bit. Fluent Bit Fluent Bit is the lightweight sibling of Fluentd. It is written in C, uses fewer resources, and is a great fit for running as a DaemonSet in Kubernetes to ship pod logs. Fluent Bit also enriches logs it collects from pods in Kubernetes using a built-in filter called kubernetes, which adds the following information: Pod Name Namespace Container Name Container ID Pod ID Labels Annotations The first four are collected from the pod tag and the last three from the Kubernetes API server. The data is stored in Fluent Bit’s cache, so there isn’t a big overhead on the API server. These records are appended to each log collected from the pods, making them easier to search. ...

This is a short post on log rotating AWS Elasticsearch indices easily using curator In the past I’ve made a simple docker container with curator and with a simple actions file for managing indices. This worked great on my own installation of Elasticsearch that managed our logs on a small scale. The actions file was: --- actions: 1: action: delete_indices description: >- Rolling indeces according to filters options: ignore_empty_list: True #disable_action: True disable_action: False filters: - filtertype: pattern kind: prefix value: filebeat- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: ${DAYS_TO_KEEP:14} And it was controlled in the docker run command, which host to work on and what old indices should be deleted, with environment variables. ...