A Simple way to do log rotate on an AWS Elasticsearch using AWS Lambda

This is a short post on log rotating AWS Elasticsearch indices easily using curator

In the past I’ve made a simple docker container with curator and with a simple actions file for managing indices.

This worked great on my own installation of Elasticsearch that managed our logs on a small scale. The actions file was:

---
	actions:
	  1:
	    action: delete_indices
	    description: >-
	      Rolling indeces accoring to filters
	    options:
	      ignore_empty_list: True
	      #disable_action: True
	      disable_action: False
	    filters:
	    - filtertype: pattern
	      kind: prefix
	      value: filebeat-
	    - filtertype: age
	      source: name
	      direction: older
	      timestring: '%Y.%m.%d'
	      unit: days
	      unit_count: ${DAYS_TO_KEEP:14}

And it was controlled in the docker run command, which host to work on and what old indices should be deleted, with environment variables.

docker run --rm ES_HOST=elastic_host -e DAYS_TO_KEEP=days_to_keep omerha/docker-elasticsearch-curator:latest

Using AWS Elasticsearch

But when using AWS Elasticsearch, even if you allow access to the elasticsearch domain completely, and using REST to access it and every thing works, using the same curator docker image will give you an authentication error. Amazon use IAM to control access elasticsearch and curator creates an elasticsearch client that requires authentication.

So, curator as is stopes working due to authentication issues.

Well, the solution is very simple, and requires couple lines of codes and move curator to work as a lambda function, that easily authenticate and does log rotate.

The trick is in the start of the function code using boto for retriving credential token, and then use the token in AWS4Auth object when building the elasticsearch client.

The git repo is https://github.com/omerh/curator-lambda-aws-es it requires you to create a new lambda, with python 3.7 runtime. With the lowest resources needed for run, and put the timeout to the max.

As explained in the README.md file, the Lambda need to in addition to the basic lambda stuff, access to elasticsearch domain

...
{
      "Effect": "Allow",
      "Action": [
        "es:ESHttpPost",
        "es:ESHttpGet",
        "es:ESHttpPut",
        "es:ESHttpDelete"
      ],
      "Resource": "arn:aws:es:*:*:*"
}
...

Using cloudwatch event you can trigger it, I use rate(1 day) and the elasticsearch host and what indices to delete are controlled with the environment variables ES_HOST and DAYS

Note, that the index time string should be timestring='%Y.%m.%d' as it is the default for all major log shippers: metricbeat, filebeat, fluentd, fluent-bit, logstash and can be easily edited in the lambda function if you changed it for any reason.

That’s it. Short and simple.

If you want to here about log shippers, which one’s to use, leave a comment bellow and I’ll do my best to answer or post about it

Using AWS Elasticsearch#

Using AWS Elasticsearch