Hi there 馃憢

Welcome to my blog, where I try to write and share about tech, and my views on tech. I mostly will talk on Kubernetes and Machine learning. About me

Splitting Kubernetes Logs by Namespaces With Fluent Bit

Today, One of the easiest ways to do log shipping from a Kubernetes cluster is by using fluent bit. Fluent Bit Fluent Bit is like the little brother of fluentd and is written in C and takes less resources, so it is the best fit for running as a Daemonset in Kubernetes for log shipping pod logs. Fluent bit also enriches the logs it is collecting from pods in kubernetes using a built-in filter called kubernetes which enriches the logs with the following information:...

May 5, 2020 路 5 min 路 854 words 路 Omer

Running jvm in kubernetes

Recently I鈥檝e joined a project with real time machine learning inference. The project was set to run on Kubernetes 1.12 on AWS while development and training was made on premise, and some were made using Docker. When the application was deployed to production we started to see poor performance and started to investigate. and it was weird, on the developers machines and in docker everything worked faster. After debugging it for a while I understood that the jvm doesn鈥檛 see all available cores inside the pod....

March 1, 2020 路 3 min 路 574 words 路 Omer

Working With AWS ECR on Kubernetes Running on Docker for mac

When you choose to run your kubernetes cluster on AWS, there are 2 easy ways: The almost valina way using Kops The amazon way using EKS The common thing between both ways, is using Amazon ECR for storing the docker images and have a worry free push, pull to and from Amazon ECR which requires a IAM Role that allows the worker nodes pulling the images saftly. Amazon ECR Amazon Elastic Container Service is one of the cheapst ways to store docker images and safer due to the nature of Amazon IAM....

August 27, 2019 路 3 min 路 512 words 路 Omer

A Simple way to do log rotate on an AWS Elasticsearch using AWS Lambda

This is a short post on log rotating AWS Elasticsearch indices easily using curator In the past I鈥檝e made a simple docker container with curator and with a simple actions file for managing indices. This worked great on my own installation of Elasticsearch that managed our logs on a small scale. The actions file was: --- actions: 1: action: delete_indices description: >- Rolling indeces accoring to filters options: ignore_empty_list: True #disable_action: True disable_action: False filters: - filtertype: pattern kind: prefix value: filebeat- - filtertype: age source: name direction: older timestring: '%Y....

July 16, 2019 路 2 min 路 417 words 路 Omer

Lambda continuous delivery using docker and Jenkins pipeline

Using AWS Lambda has become very popular. Most of the time using it is fast and just (but not always). When you start having a lot of lambda functions the complexity of managing them can be hard. When a developer can just edit the code inline, it can become the worse manage service use ever chose. So again, the most important thing to do when you start working with Lambda is to have a proper way to deliver the code from git to aws....

June 27, 2019 路 4 min 路 703 words 路 Omer

The state of continuous integration using docker

Docker has become the preferable way and the most easy way to run continuous integration, due to its reproducible nature and fast learning curve. There are multiple ways approaching CI processes using docker. Using docker build, which is the easiest way FROM node:12 ARG NODE_ENV=production COPY . /src RUN npm install WORKDIR /src CMD [ "node", "app.js" ] Using docker multi step build, this usually helps making the final image smaller and safer without all the build binaries FROM maven AS Builder COPY ....

June 2, 2019 路 5 min 路 964 words 路 Omer

Accessing EKS API Server from AWS EC2 instance

In this post I am going to describe how to maintain access to Amazon AWS EKS - kubernetes cluster with only attaching an IAM Role as an instance profile, without configuring access keys on the ec2 instance. The benefit of course is not storing any Amazon IAM credentials on the ec2 instance. Having your infrastructure more secure. In my use case, I wanted to have my Jenkins CI server have access to EKS cluster for adding continuance delivery using Jenkins declarative pipeline, to my continuance integration process....

May 28, 2019 路 4 min 路 785 words 路 Omer

My first post

This blog will run using hugo static site generator project. I am using the theme hugo-cactus-theme by digitalcraftsman which hopefully I wont edit it, its the cleanest, simplest and SEO friendly that I found. The entire blog will be made by using markdown language. The blog will be hosted on Netlify. Netlify offers CI/CD, CDN, Let鈥檚 Encrypt SSL and an optional custom domain for free. The code will be saved on Github for free....

May 17, 2019 路 2 min 路 424 words 路 Omer

0 min 路 0 words 路 Omer