Kubernetes Logging in Scalyr

Kubernetes logging is straightforward for the most part. If you haven’t yet, please read the introduction level post of Get Started Quickly With Kubernetes Logging and Getting Kubernetes Logs to Scalyr The Easy Way.

Containers typically write all their logs to the stdout so you can use
kubectl logs to explore those logs. In Scalyr, with the help of daemonset agents, you would get these logs nicely tagged, signed and delivered without any manual intervention.

Here is the reference from Kubernetes Logging Architecture document that I am going to use here. We are going to dig into how we can get the logs that aren’t being written to stdout.

Life as Devops is not as straightforward in real workloads. As you deploy enough containers in Kubernetes,  you will come across instances where some of the application logs from your deployments just don’t show up in Scalyr.

This can happen for many reasons. One of the most common one is that the logs in the container are being written to the local container volumes which is ephemeral. When you run kubectl logs <deployment/controller> and you don’t see the logs, it means you won’t see them in Scalyr either. This is expected.

I recently encountered a similar situation with several application logs being written to the Container volume at a particular file path. So I reached out to Scalyr engineering teams who are the most familiar with Kubernetes for the right way to get at these logs into Scalyr. The solution was simple and straightforward. Thanks to Steven Czerwinski, CTO and Co-founder of Scalyr for providing an in-depth guide on this.

Let’s say your target container is writing logs to a local file at /opt/log/myapp/error.log and not to stdout. There are three things we need to do to accomplish our goal of extracting logs.

  1. We need to mount a volume that contains the log file or log directory. This volumeMount needs to be shared between your primary container and the sidecar container.
  2. We need to have one sidecar container for each log file that needs to be piped to stdout. That is all this container needs to be able to do.
  3. Define this volume with type emptyDir so it’s writable by our target container.

Here’s an example of the configuration

...
spec:
  containers:
  - name: target
     image: myapp:latest
     volumeMounts:
     name: my-target-logs-dir
     mountPath: /opt/log/myapp/error.log
   - name: sidecar-1
	   image: busybox
	   args: [/bin/sh, -c, 'while ! tail -n+1 -F /opt/log/myapp/error.log  ; do sleep 1 ; done']
     volumeMounts:
       name: my-target-logs-dir
       mountPath: /opt/log/myapp/error.log
.....
   volumes:
   - name: my-target-logs-dir
     emptyDir: {}

Scalyr always has some useful tips to share in the documentation, so here’s a few more tips:

  • If you have multiple files in the same directory then mount the directory itself instead of the individual file. 
  • Make sure logs are always in their own directory and not with other things like application code and configuration. This is just good practice.  
  • Use multiple sidecars for each log file if you have more than one log file in that container. You can mount more volumes for different paths as well.
  • Give sidecars a name that identify the logs as it will come in handy when using scalyr specific annotation for setting the parser name, redaction, sampling etc.

Now that you can get the logs for complete visibility, dive deeper into Configuring Scalyr Agent (Kubernetes). First thing would be to set a custom parser for your application logs. The Kubernetes logs are always automatically parsed.