From console.log to Kibana: How Your Pod's Logs Actually Reach Elasticsearch

Almost every backend engineer has heard of the ELK stack. Elasticsearch stores the logs, Kibana lets you search them, and something in the middle ships them there.

But there is one question that confused me for a long time. Your code runs inside a container, inside a pod, on some node in a cluster that you will probably never SSH into. So how does a single console.log("payment failed") actually reach Elasticsearch?

There is no magic here. It is a simple four hop pipeline:

Your app (stdout/stderr)
   ↓
Container runtime writes logs to a file on the node
   ↓
Fluentd (a DaemonSet running on every node) tails that file
   ↓
Fluentd ships the log to Elasticsearch  →  Kibana reads it

In my DevOps Essentials post I had promised that I will cover this DaemonSet based logging pattern in detail. So this is that post. Let’s walk through each hop slowly.

---

Hop 1: Your app only writes to stdout, nothing else

The most important rule in container logging is also the one that feels the most odd in the beginning:

Your application should not manage log files. It should only write to stdout and stderr.

No app.log. No log rotation logic. No shipping code inside your service. Just print.

// This is all your app needs to do.
console.log(JSON.stringify({ level: "info", msg: "payment processed", orderId: 123 }));
console.error(JSON.stringify({ level: "error", msg: "gateway timeout", orderId: 456 }));

This is one of the 12-Factor App principles, treat your logs as event streams. Your app is just a producer that throws events out into the void. It does not know where they finally end up, and that is exactly the point. Routing, storing and searching the logs is somebody else’s job.

Why does this matter? Because it keeps your app separate from your logging setup. You can swap Elasticsearch for Loki, or Fluentd for Fluent Bit, and you do not have to touch a single line of your application code.

One small tip, log as structured JSON instead of plain strings. {"level":"error","orderId":456} is very easy to filter in Kibana. "error processing order 456" will only force you into fragile regex later.

---

Hop 2: The container runtime writes stdout into a file

So your app printed something to stdout. Where does that stream go?

When a container writes to stdout/stderr, the container runtime (Docker, containerd, CRI-O) picks up those streams and writes them into a log file on the node’s filesystem. With Docker’s default json-file logging driver, every line becomes a JSON object like this:

{"log":"{\"level\":\"error\",\"msg\":\"gateway timeout\"}\n","stream":"stderr","time":"2026-07-18T10:22:01.5Z"}

Notice how the runtime wraps your actual log line inside its own envelope. It adds which stream it came from (stdout or stderr) and a time stamp.

On a Kubernetes node these files live in some predictable places:

/var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log
/var/log/containers/<pod>_<namespace>_<container>-<id>.log   # symlinks into the above

That /var/log/containers/ directory is the important one. Every container on that node dumps its logs there, and the filename itself contains the pod, namespace and container name. Keep this point in mind, it becomes useful in the next hop.

This is also exactly what kubectl logs <pod> reads. That command is not doing anything fancy, it is just showing you the content of these node local files. And that is also why kubectl logs loses the history when a pod gets rescheduled or the node’s logs rotate. These node local files are temporary. That is the whole reason we need to ship them somewhere permanent.

---

Hop 3: Fluentd, one collector per node running as a DaemonSet

Now the real question. Who reads all those /var/log/containers/*.log files and forwards them?

A log collector. The two names you will hear the most are Fluentd and its lighter cousin Fluent Bit. Their job is to tail the log files, parse them, enrich them and push them to some destination.

But here is the tricky part. How do you make sure the collector can see the logs of every container on every node? You cannot run it as a normal Deployment. A Deployment might place 3 replicas on a 10 node cluster, and then 7 nodes worth of logs will never get collected.

This is exactly the problem that a DaemonSet solves.

A DaemonSet makes sure that one copy of a pod runs on every node in the cluster. Add a new node and Kubernetes automatically schedules the collector on it. Remove a node and the collector goes away with it.

That is exactly the guarantee that logging needs. One Fluentd pod per node, and each one is responsible only for the logs of the containers on its own node.

        Node A                    Node B                    Node C
 ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
 │  app pods...      │     │  app pods...      │     │  app pods...      │
 │  /var/log/        │     │  /var/log/        │     │  /var/log/        │
 │  containers/*.log │     │  containers/*.log │     │  containers/*.log │
 │        ▲          │     │        ▲          │     │        ▲          │
 │   ┌────┴─────┐    │     │   ┌────┴─────┐    │     │   ┌────┴─────┐    │
 │   │ fluentd  │    │     │   │ fluentd  │    │     │   │ fluentd  │    │
 │   │(DaemonSet│    │     │   │(DaemonSet│    │     │   │(DaemonSet│    │
 │   │   pod)   │    │     │   │   pod)   │    │     │   │   pod)   │    │
 │   └────┬─────┘    │     │   └────┬─────┘    │     │   └────┬─────┘    │
 └────────┼──────────┘     └────────┼──────────┘     └────────┼──────────┘
          └──────────────────────────┼──────────────────────────┘
                                      ▼
                              Elasticsearch

But how does a Fluentd pod read files that belong to the node and not to itself? Through a hostPath volume. The node’s /var/log directory is mounted straight into the Fluentd container:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels: { app: fluentd }
  template:
    metadata:
      labels: { app: fluentd }
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-elasticsearch
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "elasticsearch.logging.svc.cluster.local"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"
          volumeMounts:
            - name: varlog
              mountPath: /var/log            # the node's log dir, read-only
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

Once it can see the files, Fluentd does three things:

1. Tail, it follows every /var/log/containers/*.log file and picks up new lines as they get written (and it remembers its position, so that a restart does not resend everything again).
2. Enrich, remember the filename contains pod/namespace/container? Fluentd reads that and attaches Kubernetes metadata like pod_name, namespace, labels, node_name. So in Kibana you can filter by namespace: payments or pod: checkout-7f9c.
3. Buffer and forward, it batches the records into a buffer and ships them ahead. If Elasticsearch is slow or down, the buffer holds the logs and retries instead of dropping them.

A trimmed down Fluentd config for this flow looks like this:

# 1. TAIL — read every container log file on this node
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kube.*
  <parse>
    @type json
  </parse>
</source>

# 2. ENRICH — add pod/namespace/labels from the K8s API
<filter kube.**>
  @type kubernetes_metadata
</filter>

# 3. FORWARD — batch and ship to Elasticsearch, buffering on failure
<match kube.**>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true          # daily indices: logstash-2026.07.18
  <buffer>
    flush_interval 5s
    retry_max_times 10
  </buffer>
</match>

---

Hop 4: Elasticsearch stores it and Kibana shows it

Fluentd pushes every enriched record to Elasticsearch over HTTP. Elasticsearch indexes it, usually into a daily index like logstash-2026.07.18 (that is what the logstash_format true line above does). Daily indices make retention very easy, to delete last month’s logs you just drop those indices.

Then Kibana connects to Elasticsearch and gives you the search UI. So now that console.log("payment failed") from Hop 1 is:

• searchable by full text (msg: "payment failed")
• filterable by the metadata that Fluentd added (namespace: payments AND level: error)
• and you can correlate it across every pod and node in the cluster, all in one place

The full journey, end to end:

console.log(...)                        ← your code, Hop 1
   ↓ stdout
/var/log/containers/checkout-*.log      ← container runtime, Hop 2
   ↓ tailed by
fluentd DaemonSet pod on that node      ← Hop 3
   ↓ enrich + buffer + HTTP POST
Elasticsearch  (index: logstash-2026.07.18)
   ↓
Kibana  ← you, searching at 2am during an incident

---

The “L” in ELK is not always Logstash

One naming confusion that is worth clearing up. ELK stands for Elasticsearch, Logstash, Kibana. But in Kubernetes you will rarely see Logstash sitting on the nodes. Logstash is heavy (it runs on the JVM) and it was built as a central processing pipeline, not as a per node agent.

So the modern per node collector is almost always Fluentd or Fluent Bit:

	Fluentd	Fluent Bit
Written in	Ruby (+ C core)	Pure C
Memory footprint	~40 MB+	~1 MB
Plugins	Huge ecosystem	Smaller, growing
Typical role	Aggregator / node agent	Lightweight node agent

A very common production setup is to run Fluent Bit as the tiny per node DaemonSet and have it forward to a central Fluentd aggregator that does the heavy parsing before Elasticsearch. Same pipeline shape, just split into two tiers. (When people say “EFK stack”, that F is exactly this Fluentd/Fluent Bit swap.)

---

Some gotchas I have hit (so you don’t have to)

• Do not write logs to a file inside the container. If your app writes to /app/logs/app.log, the collector (which only watches stdout) will never see it, and that file dies with the pod. Just print to stdout.
• Multiline stack traces get split. Each line of a Java or Node stack trace is a separate stdout write, so the collector treats each line as a separate log record. You need a multiline parser to stitch them back into one event.
• Log rotation is real. The kubelet rotates the container log files (default around 10 MB). A good collector follows the rotation using its position file, a naive one can double send or miss lines around the rotation boundary.
• Backpressure will bite you. If Elasticsearch cannot keep up, the collector’s buffer fills up. Size your buffers and set retry limits, otherwise a slow Elasticsearch can OOM your logging pods or silently drop logs.
• kubectl logs is not your log store. It reads node local files that vanish on reschedule. It is good for a quick debug, but never for history or auditing. That is Elasticsearch’s job.

---

Wrapping up

The ELK stack feels like a black box until you trace the one path that actually matters:

Your app prints to stdout, the container runtime writes it to a file on the node, a Fluentd DaemonSet (one per node) tails that file, enriches it with pod metadata and ships it, Elasticsearch stores it, and Kibana lets you search it.

The nice part here is the separation of concerns. Your application does the simplest possible thing, it just prints. The DaemonSet makes sure there is a collector on every node with zero per app wiring. And swapping any piece of this pipeline never touches your code.

That is the whole trick really. No magic, just stdout, a file, and a collector that runs everywhere.