The ingest path has no message queue, and does not need one

The reflex when designing an ingest path is to put a broker in front of the database. Kafka, RabbitMQ, Redis Streams, something. We did not, and this is the arithmetic behind that.

What the load actually is

One air quality sensor reporting every metric it has produces about 1,225 rows an hour. That is 0.34 rows per second. Per sensor.

With agents batching on a 60-second timer:

| Sensors | Requests/second | Rows/second | |---:|---:|---:| | 1,000 | 17 | 340 | | 10,000 | 170 | 3,400 | | 100,000 | 1,700 | 34,000 |

ClickHouse ingests 34,000 rows a second without noticing. It is a column store built for exactly this. At the scale where this project would be a resounding success, the database is not the bottleneck and a queue in front of it would be smoothing a load that is already smooth.

The failure mode that is real

ClickHouse does not fall over from row volume. It falls over from insert frequency. Every insert creates a part; parts have to be merged; too many small inserts and the merge scheduler falls behind, at which point queries slow down and the problem compounds.

So the thing worth defending against is ten thousand agents each sending one row.

Two layers already prevent it, neither of which is a broker:

1. 1.The agent batches. Readings accumulate for 30–60 seconds and go as one request.
2. 2.`async_insert` batches server-side. ClickHouse buffers inserts and flushes on 10 MB

or 200 ms, whichever comes first.

We measured the second one rather than trusting it: twenty concurrent single-row inserts produced two parts, not twenty. The coalescing works.

The other reason people reach for a queue

Durability. If the database is down, a broker holds the writes.

But look at where the data already is. The agent has the readings on disk, in its own buffer, and it does not delete them until the server confirms they were stored. On a 5xx it keeps them and retries with backoff.

The agent fleet is a distributed, durable queue. It has properties an in-cluster broker does not:

• It survives the entire cluster being down, including the broker.
• Its capacity grows with the number of contributors, which is the same thing that grows

the load.

• It costs nothing to run, because it is running on someone else's Raspberry Pi doing

nothing else.

• Readings carry the timestamp they were taken, so a three-day outage produces a

three-day gap that fills in correctly afterwards rather than a pile of readings all stamped with the recovery time.

We tested that last part by killing the ingest service, collecting readings, restarting the agent process entirely, then bringing ingest back. All four readings arrived with their original timestamps.

When we will add one

Not on a hunch. When one of these is true:

• Fan-out. A second consumer — alerting, downsampling, a model — needs the same

stream. This is the most likely trigger, and it is the one a queue is genuinely for.

• Clients that cannot buffer. A browser submitting a one-shot reading, or a webhook

from a third-party network, has nowhere to hold data during an outage. Today every writer is an agent that can.

When that day comes it will be Redis Streams, because openaqi already runs Redis for rate limiting and the records are tiny. Kafka is a fine answer to a problem we do not have.

The general point

The queue is not free. It is a service to run, monitor, upgrade, secure and pay for, and it becomes a component whose failure takes the write path with it. Adding one because the diagram looks more serious with it is how a project acquires operational burden it never needed.

Measure first. The measurement took an afternoon and saved a permanent dependency.