← Back to all posts

My Aquarium Runs on the Same Stack as My Cluster

Title card: My aquarium runs on the same stack as my cluster. Beside the title, two step lines: the plug's state switches to ON at once, its power stays at 0 W for 29 seconds and then jumps to 49 W

TL;DR: My four aquariums are wired into the same patterns I run at work. Zigbee plugs and a Raspberry Pi called HydroSense are the exporters. A local MQTT broker next to Home Assistant, bridged into a three-node VerneMQ cluster on Kubernetes, is the event bus. Home Assistant is the reconciler: it re-asserts the light schedule every 15 minutes. Telegraf, InfluxDB and Grafana are the TSDB. This post is the map. I trace one real automation through every layer, with timestamps from the database, and then list what broke on the way.


An incident you cannot roll back

At work, a bad deploy has an undo button: Argo Rollouts keeps the previous ReplicaSet around. A heater relay that sticks on at 2 a.m. has no previous revision. Water does not roll back to 25 degrees, and the fish do not get restored from a Longhorn snapshot.

That asymmetry is why this stack looks the way it does. I wanted to stop finding out about problems by looking at the tank, so I reached for tools whose failure modes I know.

I have written about the two halves before: the hobby in Balance, the Heartbeat of a Planted Aquarium and When Work and Passion Start Speaking the Same Language, the hardware in The Aquarium Hobby: It’s Not a Philosophy, It’s a Survival Strategy, and the first data path in Unifying My Smart Home and Homelab. This post ties them together.

The map

flowchart LR
  subgraph home["At home"]
    plugs["Zigbee plugs<br/>lights, filters"] --> z2m["zigbee2mqtt"]
    z2m --> lb["local MQTT broker<br/>(Home Assistant RPi5)"]
    hs["HydroSense<br/>Pi Zero W: backlight, 2 dosers"] --> lb
    ha["Home Assistant"] <--> lb
  end
  lb --> vmq
  subgraph k8s["Cluster"]
    vmq["VerneMQ, 3 nodes<br/>pulls from the home broker<br/>over a bridge on one node"] --> tg["Telegraf"]
    tg --> mqttb[("InfluxDB<br/>bucket mqtt")]
    mqttb --> gf["Grafana<br/>Aquarium Monitoring"]
    hab[("InfluxDB<br/>bucket homeassistant<br/>history pushed by Home Assistant")]
  end
  ha --> hab

The same picture as a table, with the thing each layer is at work:

LayerIn the fish roomAt work
SignalsZigbee smart plugs via zigbee2mqtt; HydroSense on a Raspberry Pi Zero Wexporters
Event busa local broker on the Home Assistant Pi, bridged into a 3-node VerneMQ clusterKafka, any event bus
ReconcilerHome Assistant automationsan operator’s control loop
TSDBTelegraf, InfluxDB, GrafanaPrometheus, VictoriaMetrics

HydroHabitat, my own aquarium app, is in the final phase of development.

Two design decisions shape everything below.

The automations live at home, the history lives in the cluster. Home Assistant talks to its own broker on the same Raspberry Pi 5. If the cluster or the VPN link disappears, the lights still switch on at two in the afternoon. The cluster is downstream: it gets a copy of every message and is allowed to fail.

The event bus is pulled, the history is pushed. The MQTT bridge runs on the VerneMQ side and subscribes to the Home Assistant broker over the VPN, so the broker at home does not know the cluster exists. Home Assistant’s InfluxDB integration, though, writes state history straight into the homeassistant bucket, and the automations never wait for it.

Layer 1: signals (exporters)

An exporter does not know what it is measuring for. node-exporter reports a filesystem filling up, with no opinion about the database on it. A Zigbee smart plug does the same for a fish tank: it reports watts, volts, amps and link quality for whatever is plugged into it, and the meaning comes later.

The second source is HydroSense, the project that started as a drawer of old parts in the survival strategy post. Today it is a FastAPI service on a Raspberry Pi Zero W that drives the LED backlight and two Jebao dosing pumps, and publishes on hydrosense/....

The HydroSense dashboard in Home Assistant: the 112 L, 54 L and 43 L tanks, each with a display of its total power and its cost today and this month, its feed and service switches, and its filter and pump with live power

Subscribing inside the cluster shows what made it across the bridge (two messages):

zigbee2mqtt/Heater 54L {"current":0.82,"energy":1166.15,"indicator_mode":"off/on","linkquality":252,"power":190,"power_outage_memory":"on","state":"ON","voltage":236}
hydrosense/hs01/tank_54/state {"feed": false, "feed_ends_at": null, "service": false, "error": null, "temperature": null, "water_level": null, "mode": null, "power_w": 194.9, "filter": true, "filter_power_w": 4.9, "filter_energy_kwh": 7.43, "heater": true, "heater_power_w": 190.0, "heater_energy_kwh": 1166.15, "light": false, "light_power_w": 0.0, "light_energy_kwh": 400.79, "pump": true, "pump_power_w": 0.0, "pump_energy_kwh": 720.08}

The first is a raw plug report from zigbee2mqtt; the second is HydroSense folding the same plug readings into a per-tank state.

The zigbee2mqtt device list: Tuya and Aqara plugs with their model and link quality

Layer 2: the event bus (two brokers, on purpose)

There are two brokers with different jobs. The local broker on the Home Assistant Pi is the one the devices and the automations talk to. It is a single node, deliberately: it has to be as available as the Pi it runs on and no more.

The cluster broker is VerneMQ, three nodes on Kubernetes. It replaced a three-node EMQX in four pull requests merged the same evening. The bridge and Telegraf subscribe here.

Why VerneMQ, and why built from source

OptionLicenseClusteringVerdict
EMQX, last Apache-2.0 lineApache-2.0yesmy Trivy scan: 8 Critical, 141 High
EMQX, newer releasesBSL 1.1: production use of a single node is free, a cluster needs a license filewith a license fileno
VerneMQ packages and Docker imagesVerneMQ EULA, not applicable to commercial useyesnot allowed here
VerneMQ built from sourceApache-2.0yeschosen
NanoMQ, MosquittoNanoMQ is MIT; for Mosquitto the license is not the issueno clustering in either core that I foundno

So I build VerneMQ from source through my usual pipeline: Trivy scan and cosign signature. Trivy reported 0 Critical, High or Medium on the local test build. Clustering is what I am paying for, and building from source is the price.

The bridge runs on one node only

The pull bridge runs on one node and nowhere else, because one bridge per node would deliver every message three times. When that node restarts, the house broker keeps QoS 1 messages for the bridge’s session and they arrive late, not never.

Layer 3: Home Assistant (the reconciler)

A Kubernetes controller compares desired state with observed state, acts on the difference, and does it again on a timer, so a missed event or a manual change gets corrected on the next pass. Home Assistant runs my aquarium lights that way. Two automations, turn_on_aquarium_lights and turn_off_aquarium_lights, cover the four light switches. Both trigger every 15 minutes around the clock, and a time condition decides which one acts: the on one from 14:00 to 22:00 local time, the off one for the rest of the day. A state trigger also puts a light back at once if I switch it by hand or a plug returns as off. A watch plus a resync period is how a controller is built.

alias: Turn on aquarium lights
triggers:
  - trigger: time
    at: "14:00:00"
  - trigger: homeassistant
    event: start
  - trigger: state
    entity_id:
      - switch.light_112l
      - switch.light_54l
      - switch.plant_boxes
      - switch.small_aquarium_lights
    to: "off"
  - trigger: time_pattern
    minutes: /15
conditions:
  - condition: time
    after: "14:00:00"
    before: "22:00:00"
actions:
  - action: switch.turn_on
    target:
      entity_id:
        - switch.light_112l
        - switch.light_54l
        - switch.plant_boxes
        - switch.small_aquarium_lights

turn_off_aquarium_lights is the mirror image.

The Home Assistant trace of turn_off_aquarium_lights at 9:15 in the morning: four triggers in the graph, the 15-minute time pattern fired, the time condition passed, and the switch turn-off action ran

Re-applying desired state is only safe if the controller cannot be wrong in a loop. HydroSense taught me that, and the story is below.

Both heaters have their own thermostats, and their plugs report watts like every other plug: the 54 L one cycles between 0 and about 200 W. HydroSense’s service mode can also switch them off, for a bounded time. If a thermostat fails closed, the heater stays on and nothing in the stack notices yet. The second opinion I am adding is a temperature probe per tank, with HydroSense cutting the plug above a limit. Until it exists, a Grafana alert fires when a heater’s power has not dropped to 0 W for three hours.

Layer 4: InfluxDB and Grafana (the TSDB)

Telegraf subscribes to VerneMQ with one input per topic tree (zigbee2mqtt/+ and hydrosense/+/+/state) and a persistent session, so a pod restart does not drop queued messages. It writes to InfluxDB, bucket mqtt, retention infinite, on a Longhorn volume in the daily backup set.

Home Assistant writes its own state history into a second bucket, homeassistant, which is where the automation timestamps in the trace below come from.

The Grafana Aquarium Monitoring dashboard: the Heater row shows the 112 L heater switching between 0 and about 160 W, above the Filters and Aquaponics pumps rows

The plugs also report an energy counter, so the dashboard prices them. A cost row multiplies each counter’s increase by my electricity price and breaks it down per tank, per device and per day. Over one week the whole fish room cost about 30 PLN, and the two heaters were more than half of it.

The Costs row of the dashboard: cost, energy and a 30-day estimate for the selected range, and the cost per tank, per device and per day

The alerting side is VMRules evaluated by VictoriaMetrics: VerneMQDown fires when fewer than two nodes are scraped, VerneMQClusterPartitioned when a node has seen a member down for 10 minutes, and three Telegraf* rules cover the next hop.

One automation, end to end

One real event: the light on the 112 L tank switching on at 14:00 local time, reconstructed from the two buckets (times in UTC):

Time (UTC)WhereWhat
13:00:00.384bucket homeassistantturn_on_aquarium_lights runs
13:00:00.454bucket mqttzigbee2mqtt/Light 112L reports state=ON, power=0
13:00:00.571 to 13:00:01.371bucket mqttfour more reports, all state=ON, power=0
13:00:29.578bucket mqttpower=49

The day before: the automation at 13:00:00.385, the first ON report at .452, and 48 W at 13:00:30.632.

Between the first two rows the event went through the local broker, zigbee2mqtt, the mesh and the plug’s relay, and back across the VPN to Telegraf and InfluxDB. The two timestamps come from different clocks (Home Assistant’s on the Pi, Telegraf’s receive time in the cluster), so I will not claim 70 milliseconds. I will claim “under a second”, which both days support.

The more interesting gap comes after that. The plug reported ON immediately and 0 W for almost thirty seconds. The relay closed; the measurement had not caught up. Like a pod that is Running but not ready, status and effect are different signals, and an alert that trusts the first will tell you the light is on while the tank sits in the dark.

The Home Assistant bucket holds two writes per automation run, about 3 ms apart (start and finish), and four at 14:00, where the time trigger and the 15-minute pattern fire together. The 13:00:00.384 in the table is the first run starting.

What broke before it worked

No tank overheated or flooded; what broke was the stack.

The doser kept writing to itself. One day the older Jebao pump closed any TCP session idle for about 15 seconds while the library pinged every 20, and on every reconnect HydroSense re-read the per-head calibration by switching a setting on the pump and back. Result: 36 writes to the pump in 6 minutes, on a device that should only be written when I ask. The fixes: a heartbeat of 8 seconds, below the firmware’s timeout, and no pump writes on reconnect. A reconciler that writes on every resync is just a loop with a side effect.

What works

The automations run at home and do not depend on the cluster. The history lives in the cluster, and one automation can be followed from Home Assistant to the database in under a second. The bridge lives on one node by design, and a restart delays messages rather than losing them. Precise water temperature measurements, automatic top-off, pH measurements and heater safeguards are still to come, in HydroSense (my aquarium controller) and in the HydroHabitat app.

All four tanks are running, but they are young: the nitrogen cycle has not closed yet, so ammonia and nitrite still come from a test kit. The stack is ahead of the biology.