Self-Hosted Homelab Observability: How I Monitor My Network With Proxmox, ntopng, Docker and FastAPI

I run my whole house on a homelab, and for years my monitoring was a pile of browser tabs. Proxmox VE in one, ntopng in another, OPNsense somewhere, WireGuard peers I could never quite account for, and a Docker host I only checked after something already broke. This is the article I wish I had when I started: how I turned that mess into real self-hosted monitoring and network observability, using Proxmox, ntopng, Docker, and a FastAPI backend I wrote myself. Everything here is my own personal lab and open-source tooling, so you can rebuild all of it.

The problem with tab-based homelab monitoring

A homelab grows one service at a time. You add Proxmox, then a reverse proxy, then a VPN, then a game server for friends, and each one ships its own little web UI. Individually they are fine. Together they are a blind spot. Nobody sits and refreshes eight dashboards at 2am, so you only find out a disk filled up or a tunnel dropped when a service is already down.

The fix is not another all-in-one appliance. The fix is a thin layer that pulls the state you already have into one place, keeps history, and tells you when something is wrong. I split that into two jobs: a unified operations portal, and a dedicated flow-observability pipeline for the network itself. I will walk through both.

EmberEye: one dashboard over Proxmox, ntopng, OPNsense and the rest

EmberEye is the portal I built to kill the tabs. It unifies Proxmox VE, ntopng, NGINX, OPNsense, WireGuard, Portainer, and a game panel behind a single dashboard. The stack is deliberately boring and reproducible: FastAPI for the backend, Next.js and React for the frontend, PostgreSQL for storage, Redis for caching and queues, and an NGINX reverse proxy in front. The whole thing runs in Docker, so bringing it up on a fresh host is a compose file, not a weekend.

A few design choices carried their weight:

• APScheduler background polling. Each integration is a small poller on its own interval. Proxmox node health, ntopng traffic, OPNsense gateway status, container states from Portainer. Slow sources never block fast ones.
• WebSocket real-time push. The browser does not poll the API in a loop. The backend pushes updates as they land, so the dashboard stays live without hammering anything.
• JWT auth. It is exposed only over my VPN, but auth is still non-negotiable. A homelab portal that can see everything is a target, so treat it like one.
• An alerts engine. Rules with cooldowns so I am not spammed, firing to a Discord webhook and email. Cooldowns are the difference between a useful alert and a channel everyone mutes.
• A canvas service map. A topology view of what talks to what, which is worth more than any single metric when you are trying to reason about a failure.
• Metric retention with hourly rollups and CSV reports. Raw samples for the recent window, hourly rollups for history, so the database does not grow without bound.

If you take one idea from EmberEye, take the shape: pollers write to Postgres, Redis smooths the hot path, WebSockets push to the UI, and an alert engine sits on top. That pattern generalizes to almost any home setup.

EmberScope: passive flow observability for the home network

The portal tells me if a service is up. It does not tell me what my network is actually doing. For that I built EmberScope, a self-hosted flow-observability platform, and I built it around four rules I now apply to anything that touches my network: passive, additive, reversible, and resource-fenced. Nothing I add is allowed to sit in a load-bearing path. If EmberScope fell over completely, my internet would not notice.

Here is the pipeline, and why each piece earns its place:

• Suricata as the primary sensor. I already run Suricata as an IDS on OPNsense, so instead of standing up a new firewall daemon, I extended the running instance to emit EVE flow, http, tls, and dns events. No new inline component, no new failure mode on the edge.
• softflowd on a remote edge VPS. A second vantage point emitting IPFIX biflow records, so I can see egress from outside the house, not just from behind my own firewall.
• Vector as the single ingest bus. Every sensor is messy in its own way. Vector normalizes everything into one canonical flow record, enriches with GeoIP and ASN data from MMDB files, dedupes, and disk-buffers inserts so a downstream hiccup never loses events.
• ClickHouse as the flow warehouse. Flow data is huge and append-only, which is exactly what a columnar store is for. TTL handles retention, and materialized views roll data up at one-minute, one-hour, and daily grains so queries stay fast.
• Arkime and OpenSearch for packet capture. When a flow record raises a question, I want to rewind the actual session, not guess.
• community_id as the correlation key. The same flow seen by Suricata and by softflowd shares one community_id, so stitching the two sensors together is trivial instead of a join nightmare.

The connectivity detail I am most careful about: the collector reaches the edge VPS over a dedicated WireGuard telemetry tunnel where it dials out. There are zero inbound holes punched in my network for any of this. Telemetry should never become an attack surface.

Turning data into alerts that matter

Collecting flows is easy. Knowing what is abnormal is the hard part, and it is where most homelab monitoring stops. EmberScope builds nightly baseline profiles of normal behavior, then runs a handful of detectors against them: volumetric z-score for sudden traffic spikes, new-ASN and new-country egress for a host suddenly talking somewhere it never has, DNS entropy for tunneling and generated domains, and beaconing detection for the steady heartbeat that malware and misconfigured agents both produce. A hit fires a Discord alert with a one-click pivot straight to the flow record and the captured packets.

That pivot is the whole point. An alert that just says "something is weird" wastes your night. An alert that drops you onto the exact session, with the packets, lets you make a decision in a minute.

Where the firewall automation fits

Two smaller tools close the loop. On OPNsense I automate through its API: watching gateway RTT and loss, firewall logs, and DHCP leases, and driving changes programmatically instead of clicking through the UI. EmberMonitor sits on top of ntopng and OPNsense data as an AI-assisted threat-hunting layer that can suggest and drive firewall blocks. And PortSync keeps a cloud security group and the live host iptables on a WireGuard gateway reconciled, flagging drift before it becomes a hole nobody remembers opening.

How to start your own

You do not need all of this on day one. Start with the portal pattern: one FastAPI service, Postgres, Redis, Docker, polling the two or three things you already run, pushing to a small dashboard, with an alert rule or two into Discord. Once that is stable and you trust it, add passive flow collection from a sensor you already have, like Suricata, before you reach for anything heavier. Keep every addition passive, additive, and reversible, and your homelab observability will grow with you instead of collapsing under itself.

I am Abdulmuhsen Alayouni (عبدالمحسن العيوني), and I build this out of my home in Riyadh because I would rather understand my own network than trust that it is fine. If you are building the same, I hope the map above saves you a few of the nights it cost me.