Units are the hard part

The first version of this project supported ten metrics, because that is what the one sensor on the desk reported. Expanding to cover what people actually own turned out to be less about adding rows to a catalogue and more about a problem that has sunk better datasets than this one.

The same pollutant, three ways

Nitrogen dioxide is reported by real devices as:

• ppb — parts per billion, a volume mixing ratio
• ppm — the same thing, a thousand times larger
• µg/m³ — micrograms per cubic metre, a mass concentration

The first two are trivially related. The third is not, because converting between a mixing ratio and a mass concentration depends on how many molecules are in a cubic metre — which depends on temperature and pressure.

ppb = µg/m³ × 24.45 / molar mass

24.45 is the molar volume of an ideal gas in litres per mole at 25 °C and 1 atmosphere. At 0 °C it is 22.41. The same reading converts about 9% differently depending on which reference you assume, and nobody states which one they used.

We picked the EPA convention — 25 °C, 1 atm — and wrote it into the catalogue where the conversion factors are generated, rather than leaving it implicit in a constant somebody would later "clean up".

The failure that has no signature

Temperature is worse, because the failure is silent.

A sensor reports 71.6. In Fahrenheit that is a comfortable room. In Celsius it is a sauna. Both are physically plausible, both pass any sanity check you can write, and once it is stored there is no way to tell which one it was.

This is the case that motivated putting unit on the wire:

{"metric": "temperature", "value": 71.6, "unit": "F"}

Optional, because every agent written before it existed omits it and must keep working — an absent unit asserts the value is already canonical. Present, it is converted at ingest and the canonical value is stored.

Convert before you validate

The ordering matters and it is easy to get backwards.

Each metric has a plausible range: temperature is −40 to 85 °C. If you range-check the value as received and then convert, 71.6 °F sails through the Celsius bound and is stored as a scorching room. The check has to happen on the converted value or it is worse than no check at all, because it produces false confidence.

There is a test named after exactly this, whose first assertion is that 72 passes the Celsius range check — because if that ever stops being true, the test is no longer testing what it was written for:

func TestUnconvertedFahrenheitWouldHaveBeenPlausible(t *testing.T) {
	m, _ := Lookup("temperature")
	if !m.InRange(72.0) {
		t.Fatal("premise changed: 72 no longer passes the Celsius range check")
	}
	...
}

What we can and cannot catch

The agent warns when a value is implausible as sent but sensible in another unit:

level=WARN msg="this reading is implausible as sent, but makes sense in another unit"
  metric=temperature sent=150 assumed_unit=°C looks_like=F would_be="65.56 °C"

That catches the loud half. It cannot catch 71.6, and nothing can — which is the whole argument for making the unit explicit rather than clever.

Refusing to guess

An unrecognised unit is rejected rather than passed through. That felt harsh when we wrote it, and it is right: a reading whose meaning we are guessing at becomes part of a public average that other people make decisions with. Dropping it costs one reading. Storing it costs the credibility of every number near it.

The rejection comes back to the agent with a reason, so the contributor can see what their sensor is doing wrong instead of watching data silently not appear.

Two things called TVOC

One more trap, for anyone building something similar.

"TVOC" means an index to UniFi and Sensirion — a relative scale against the sensor's own recent baseline, usually 0–500. It means a concentration to Awair and to SGP30-class parts, in ppb.

They are not the same measurement and they are not convertible. Storing both under one key would attach ppb-scale health thresholds to an index-scale reading, which is how you publish a green square in a room that should be red. They get separate keys, and the one that already had stored data kept its meaning — renaming it would have silently reinterpreted every historical row.

The catalogue is generated

Forty conversion factors typed by hand is a table with a mistake in it. Ours are computed by a script from molar masses stated once, and both the TypeScript and Go copies are written by the same run. A build check fails if they diverge, because thresholds carry health meaning and a disagreement between two languages is a correctness bug, not an inconsistency.