Home Network Flow Telemetry with Suricata and ClickHouse: My Homelab Pipeline

I run a fairly serious homelab, and for a long time I could see that traffic was happening without being able to answer the interesting questions about it. Which host talked to which ASN last night? Is that IoT device beaconing to somewhere new? What actually egressed while I was asleep? So I built my own network flow telemetry stack around Suricata, Vector, and ClickHouse, fed by both NetFlow-style IPFIX records and Suricata EVE events. This post is how it fits together and, more importantly, the design rules I refused to break while building it in my homelab.

The design philosophy: passive, additive, reversible, fenced

Before any code, I set four rules that every part of this pipeline had to obey. I am the only network admin here, and the family internet is a load-bearing path. Breaking it to satisfy my curiosity is not acceptable.

• Passive. Sensors observe. Nothing in the telemetry path is allowed to sit inline and become a new thing that can drop packets.
• Additive. I extend tools that are already running rather than introducing new daemons on the firewall. No new attack surface on the box that matters most.
• Reversible. Every phase can be turned off and leave the network exactly as it was. If a component misbehaves, I remove it and nothing else notices.
• Fenced. Every collector runs under explicit CPU, memory, and disk limits. A runaway query or a full buffer degrades the observability stack, never the router.

I built the whole thing in snapshot-gated phases. Take a snapshot, add one capability, verify the load-bearing paths are untouched, then move on. It sounds slow. It is why the internet never went down while I was instrumenting it.

Sensors: reuse Suricata, add IPFIX at the edge

My firewall already runs Suricata as an IDS. Instead of standing up a separate flow daemon on it, I extended that existing Suricata to emit EVE flow, http, tls, and dns events. That is the additive rule in practice: the daemon is already there, already inspecting, so I just turned on the event types I wanted. No new firewall service, no new failure mode on the edge.

For a second vantage point I run softflowd on my edge VPS, exporting IPFIX biflow records. Classic NetFlow/IPFIX gives me the volumetric who-talked-to-who picture from a host that sits closer to the open internet, and it complements Suricata's richer application-layer view. Two sensors, two perspectives, one canonical record downstream.

Vector as the single ingest bus

Everything lands in Vector, which I use as one normalization bus. This was the piece that made the rest sane. Suricata EVE JSON and softflowd IPFIX look nothing alike, so Vector's job is to melt them both into a single canonical flow record. In that same stage I enrich with geo and ASN data from MMDB databases, dedupe records that both sensors saw, and buffer to disk so a downstream hiccup never means lost telemetry. Vector then does batched, disk-buffered inserts into the warehouse. One place to reason about schema, one place to add enrichment, one place that absorbs backpressure.

ClickHouse: the columnar flow warehouse

Flow data is append-heavy, time-ordered, and enormous. That is exactly what ClickHouse is good at, and it is the heart of the stack. A few choices that paid off:

• TTL retention. Raw flows expire on a schedule so disk usage stays bounded without me babysitting it. The fenced rule again, expressed in SQL.
• Materialized-view rollups. I pre-aggregate into 1-minute, 1-hour, and daily rollups. Dashboards and baseline jobs read the rollups, not the raw firehose, so queries stay fast even as raw data ages out.
• Dictionaries. ASN, country, and host labels live in ClickHouse dictionaries so enrichment lookups are cheap at query time.

The result is a warehouse I can actually ask questions of interactively. Top talkers by ASN over the last hour, egress by country, per-host history, all of it returns before I lose my train of thought.

Packet rewind with Arkime and correlation with community_id

Flow records tell me that something happened. Sometimes I want the packets. Arkime backed by OpenSearch gives me full session capture and rewind for the segments I care about, so I can pivot from an interesting flow to the actual bytes.

The glue that makes cross-sensor pivoting work is community_id. Both Suricata and my flow pipeline compute the same community_id hash for a given flow, so the same connection carries one stable key across every store. A record in ClickHouse, an EVE event, and an Arkime session all line up. Without a shared correlation key, multi-sensor telemetry is just three lonely datasets.

Zero inbound holes: the collector dials out

The VPS sensor has to reach my home collector, and I was not going to open an inbound port on my network to let it. Instead I run a dedicated WireGuard telemetry tunnel where the collector dials out to the VPS. Telemetry flows back over an established tunnel, and there is not a single new inbound hole in my perimeter. This is the passive and reversible rules applied to the network edge itself.

Turning flows into alerts

Storage is not the point. Answers are. Nightly jobs build baseline profiles from the rollups, and a set of detectors run against fresh data:

• Volumetric z-score for hosts or destinations moving far outside their normal traffic band.
• New-ASN and new-country egress, because a device suddenly talking to a network it has never touched is worth a look.
• DNS entropy to surface domain-generation-style lookups.
• Beaconing detection for the steady, regular heartbeat that automated callbacks tend to produce.

When a detector fires, it sends a Discord alert with a one-click pivot straight to the offending flows and, where I captured them, the packets. That last part matters. An alert I cannot investigate in ten seconds is an alert I will learn to ignore.

Why build it this way

If you take one thing from this for your own homelab, let it be the four rules. Passive, additive, reversible, fenced turned a project that could have taken down my home network into one that quietly runs in the background and has never once caused an outage. The tool choices, Suricata for rich events, softflowd for IPFIX, Vector to normalize, ClickHouse to store, Arkime to rewind, are all replaceable. The discipline is not. Build the telemetry so that the worst it can ever do is stop telling you things, never break the thing it is watching.