Ah, logs. A bunch of information composed by lines of texts that records a small history of your application runtime. Either from printing variables to see the states, like how we caveman do debugging, up to doing audit trail because it's a business requirement. Saving any events in the form of log is not only helpful when trying to reproduce a bug or error, but also to see the usage of features in our app. In other words, logging is one way to observe our application.

To observe our application, we can go through the logs printed to console/terminal, reading from log files, or go to somewhere where the logs are stored. To observe from terminal means you need to have access to stdout of the application. It might also go to stderr but in the end, you will need to have access to those file descriptors. You do this when developing locally, where the logs are mostly ephemeral; it lasts as long as the terminal session (could be longer if it's saved, but well). You can also just transport the logs to some file, making it easier to be processed by other apps (you can open file descriptor for reading from different processes at one time). At application or service level, this is probably enough. Most of the practices will have the log stored somewhere relative with the app directory, or directly to somewhere meant to store logs (like /var/log/ in Linux).

But what if you are maintaining different services and applications? These apps can't have same access to write a file at the same time (actually you can, but simply don't). You'll end up with each services having their own log files, and complexity increases. Not mentioning log rotation and lots of other things. There are lots of hassle when dealing with logs, which makes it hard to observe and having bird's eyes view of our app.

Nowadays, there are lots of solutions to these observability measures. Enterprise, industry-grade level would be using ELK stack (Elasticsearch, LogStash, Kibana), or something even "pricey" like Datadog or Splunk. Of course, these are pricey. And it really requires considerably "good" server capacity. Your puny 1GB RAM Virtual Machine won't handle that. Some even begged for multiple node setup to achieve that "high availability". After all, everything is "distributed system" now.

I've tried ELK stack on my personal laptop, with Core i7 processor and 40GB of RAM. And running a "modest" ELK stack managed to give me BSOD thrice. Granted, it was in Windows and through WSL2. While it really raises unnecessary urgency for me to install native Linux, I searched for other solution, which you already know from the title. From the CNCF itself, we have Grafana, with it's various stacks also.

Grafana is tied with Prometheus most of the time. You might have heard Prometheus before, which simply an external application to monitor the metrics of your services. Either from scraping or actively receiving data, it serves as a solution to monitor your services. Either monitoring the VM (nodes) or even the containers. This become more apparent with how Kubernetes works, and you just need better observability. But the main actor here is Grafana, which also come with various stacks. People know Grafana as a way to better visualize the collected metrics, giving you ability to design dashboards and further process data.

Today, I am more concerned with the logs produced by my services. All I want is for them to send the logs to a centralized log store, and then query them later on. As it focuses on logs, Grafana has Loki, where the (kinda) sole purpose is to save your logs.

So what do I need? Grafana, and Grafana Loki.

Setting Up

When people say Docker is a black magic, this might be one good case of it. Setting them manually is a hassle, like I don't even want to think about it. I instantly go to Docker Hub and search for Grafana Loki and Grafana images. Found some, and added docker-compose.yaml in my project.

tl;dr; my docker compose services are like this

version: "3"


networks:
    loki:


services:
    loki:
        image: grafana/loki:2.8.0
        ports:
            - "3100:3100"
        command: -config.file=/etc/loki/local-config.yaml
        networks:
            - loki


    grafana:
        environment:
            - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
            - GF_AUTH_ANONYMOUS_ENABLED=false
        entrypoint:
            - sh
            - -euc
            - |
                mkdir -p /etc/grafana/provisioning/datasources
                cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
                apiVersion: 1
                datasources:
                - name: Loki
                  type: loki
                  access: proxy 
                  orgId: 1
                  url: http://loki:3100
                  basicAuth: false
                  isDefault: true
                  version: 1
                  editable: false
                EOF
                /run.sh
        image: grafana/grafana:latest
        ports:
            - "3000:3000"
        networks:
            - loki

Now, this looks pretty complex. As you can see we have 2 services running: Grafana and Loki. One is for visualizing the data and one is to store the logs (guess which is which). The definition for Loki will use built-in local configuration, where it will be set to run as a standalone cluster, with other configs. The detail on the file contents can be found here. On the other hand, Grafana has longer lines, mainly because we're setting the entrypoint data inline. As you can see, it will use Loki from the local network (hence the URL) as one of the data sources. You can add external data source if you want through the web dashboard as well.

After that, just type up docker compose up -d to run all the services (in detached mode) and you're good to go!

Pushing Logs with Winston

I am using Nodejs as my main Javascript runtime, and I've been relying to winston as a package to handle logs. It has so much resources related to it and many community-made transports and formats. I am not in the mood of making one by myself now, so I just searched for any Winston and Loki transport. And of course it exists!.

If you've used winston before, you already know that adding the transport is as simple as adding it in the createLogger options. I did mine like this.

        new LokiTransport({
            host: process.env["LOKI_HOST"]!,
            basicAuth: ${process.env["LOKI_USER"]!}:${process.env["LOKI_PASS"]!} ,
            batching: false,
            format: winston.format.combine(format.timestamp({ format: "YYYY-MM-DD HH:mm:ss:ms" }), format.json()),
            level: "debug",
            labels: {
                app: "closure",
                env: process.env["NODE_ENV"] === "development" ? "dev" : "prod",
            },
        }),

I still label my Izuna bot with closure tho. You can see that you'll need authentication in the form of username:password (because it's Basic auth scheme), but it's because I connected it to my Loki instances in Grafana Cloud. Using local deployment won't need the auth so you can just skip.

Run you app, and hope for Loki to receive the pushed logs.

Querying Logs

To check if you're actually pushing logs, you can open Grafana and go to the left side panel (side bar?).

And click on Label Browser button. If you can see some labels, then your logs are received successfully.

You can create queries to further fetch and filter the logs, and further creating great views and dashboards using the collected data! For example, I created this simple dashboard for my discord bot. Very simple, but it's the information I needed for now.

Those that can do better data visualizatoin will love the flexibility of Grafana.

Afterwords

Now, this is just the "beginning". I've just managed to add log aggregation system to my apps where I can see my logs better. This covers the log section, but it's still far from best practices of Observability. Sometimes, we want to know if the server is overloaded; CPU hits the upper cap, RAM gets full, or run out of storage. I'll cover the "metrics" part of observability in the next post.

Until next time, see ya.

Beginning Observability with Grafana