As Designed

How Emberline Works

65 million NFIRS fire & EMS incident records (1980–2025), processed to Parquet on Cloudflare R2 and queried in-browser via DuckDB-WASM — with zero backend cost for per-department queries.

NFIRS SOURCE65M recordsEXTRACT.dbf + .txtTRANSFORMcanonical mapVALIDATEDuckDB checkR2 + DUCKDB-WASMservesourceetletletlserving
65M records · 45 years (1980–2025) · ~70% national coverage · $0 per dept query
data caveats
Designed to be honest

What the data can and can't say

NFIRS is voluntary — roughly 70% of U.S. departments participate. We treat these caveats as first-class metadata, not footnotes. They travel with every query and surface persistently in the UI rather than being buried in a disclaimer.

Not a census

NFIRS is voluntary. Raw counts are never presented as authoritative state or national totals. Every aggregate carries an “estimate” badge and a coverage note. The UI enforces this — there is no opt-out.

Scope changes across 45 years

What NFIRS collected changed twice. EMS data is absent 1980–1998 and again 2004–2013. Call-composition charts are gated behind source_release_scope metadata stamped at ETL time — the UI cannot render a donut for those windows.

Format & key discontinuities

2012: format shifted from dBASE to delimited text. 2018/19: the NFIRS incident key was dropped — UIDs are now synthesized by hashing STATE + FDID + INC_DATE + INC_NO + EXP_NO. Trend lines crossing these years show the break explicitly.

Data Pipeline · As Designed

Why no database server?

NFIRS was retired in January 2026 — the corpus is frozen. All computation happens at ETL time or in the browser. There is no running database server in production. Pre-computing aggregates once is cheaper and faster than serving a live query engine.

EXTRACT
Format-Aware Extraction
  • Pre-2012: dBASE .dbf files
  • 2012+: delimited .txt files
  • ~20 NFIRS linked tables
WHY NFIRS changed formats mid-series. Isolating format logic in separate extractors keeps it out of the canonical pipeline — only the extractor changes if a new source format appears.
TRANSFORM
Canonical Mapping
  • incident_uid hash synthesis
  • Vintage-aware code decoding
  • 1:many modules → STRUCT[]
WHY All NFIRS quirks — code sets, key schemes, vintage differences — live here. Everything downstream is source-agnostic. This is the only layer that changes when NERIS lands.
VALIDATE
DuckDB Self-Check
  • Row count vs. FEMA targets
  • incident_uid uniqueness
  • Category distribution audit
WHY We validate with the same engine that serves. If DuckDB can't read it back cleanly, production won't either. Build fails on any FAIL exit code.
LOAD
Parquet + ZSTD to R2
  • Parquet + ZSTD level 5
  • Partitioned: year / state
  • Sorted: fdid + incident_date
WHY Sorting by fdid+date within partitions — not a separate fdid partition — enables row-group pruning without creating thousands of tiny files.
Data Coverage · As Designed

45 years is not one dataset

NFIRS changed its collection scope twice over 45 years. The frontend is built to never smooth over these breaks — scope bands travel as metadata from ETL to every chart. Hover the timeline to explore.

1980198519901995200020052010201520202025200420122019
Hover to explore scope by year
Fire only
All-services
Fire / hazmat only
Discontinuity
Serving Stack · As Designed

Zero server cost for common queries

Shape A queries — single-department detail — run entirely in DuckDB-WASM in the user's browser. R2 SQL is a fallback for genuinely novel cross-department queries, not the primary path. The frozen corpus makes pre-computation cheap and reliable.

Storage
Cloudflare R2
  • Partitioned Parquet files
  • Iceberg catalog (for R2 SQL)
  • Pre-computed aggregates
$0.015 / GB-mo
In-Browser Compute
DuckDB-WASM
  • ~33 MB, lazy-loaded
  • Never blocks first paint
  • Reads R2 directly via httpfs
  • Row-group pruning on fdid
zero backend compute
Reference
Cloudflare KV
  • Department names & FDIDs
  • NFIRS code lookups
  • Vintage code mappings
~$0 / lookup
R2 SQL — fallback only

Novel or national queries that can't be served from pre-computed aggregates fall back to R2 SQL — Cloudflare's managed query engine at ~$2.50 per TB scanned. This is the exception. Pre-computation handles 95%+ of queries at zero cost.

Query Routing · As Designed

Every query is classified before it runs

Shape A queries never leave the browser. Shape B queries check pre-computed aggregates first, then fall back to R2 SQL only if needed. Click a query type to trace how it routes.

Try a query type
query enters hereSHAPE ROUTERDUCKDB-WASMShape A · in browser$0R2 SQLShape B · fallback~$2.50 / TBCLOUDFLARE KVReference · lookup~$0
Select a query type above to trace how it routes.
Storage Layout · As Designed

What's actually in R2

Row-group pruning, not partition explosion. We partition by year and state only, then sort within partitions by fdid + incident_date. DuckDB can skip entire row groups without downloading them — a typical per-department query fetches under 50 MB from a 100+ GB state partition.

R2 Bucket emberline-nfirs
year=2024 expanded ↓
state=PA
0000.parquet
0001.parquet
state=CA
0000.parquet
0001.parquet
state=TX
0000.parquet
0001.parquet
state=NY
0000.parquet
0001.parquet
state=FL
0000.parquet
0001.parquet
state=IL
0000.parquet
0001.parquet
state=OH
0000.parquet
0001.parquet
state=GA
0000.parquet
0001.parquet
+43 states
sorted by fdid + incident_date within each file → enables row-group pruning
1980199020002010201520202021202220232025 +36 more
Pre-computed aggregates agg/
dept_year_incidenttype.parquet per-dept per-year rollup
state_year_incidenttype.parquet national / state-level rollup
dept_incidenttype_alltime.parquet lifetime totals per dept
46 years × ~50 states × ~12 files/state ≈ 27,600 Parquet objects

The sort order is the key insight. DuckDB reads only the row groups where dept_fdid matches the query, skipping the rest without downloading them. A separate fdid partition would create file-count explosion; sorting within state partitions achieves the same pruning at a fraction of the object count.

Incident Data Dictionary · Source Compatibility

Canonical fields carry source-system metadata

Every displayed canonical field is tagged by current source compatibility: NFIRS-backed, expected to map to NERIS, derived by Emberline, or not yet confirmed. This follows the canonical schema in 03-data-model.md and the adapter-seam strategy in 04-neris-transition.md.

NFIRS-backedExpected NERIS mapDerived by EmberlineNot yet confirmed
Canonical fieldStatusDisplay meaningCompatibility note
source_systemDerived by EmberlineOrigin system for each canonical row.Stamped as NFIRS today and becomes the split key for future NERIS rows.
incident_uidDerived by EmberlineStable canonical incident identifier.Synthesized from source keys so NFIRS and NERIS can share one serving shape.
dept_fdid / state / dept_nameNFIRS-backedReporting department identity and geography.Backed by NFIRS FDHeader-era identifiers; later NERIS department crosswalk remains to be confirmed.
incident_date / yearExpected NERIS mapNormalized incident date and year partition.NFIRS-backed today; expected to map directly as the shared trend axis.
alarm_ts / arrival_ts / cleared_tsExpected NERIS mapResponse-time timestamps for duration analysis.NFIRS-backed where present; expected NERIS operational times need adapter validation.
incident_type_code / labelNFIRS-backedRaw source incident type and decoded source label.NFIRS-specific code vocabulary; NERIS should not leak its raw equivalent into frontend queries.
incident_categoryExpected NERIS mapSource-agnostic kind: fire, EMS, hazmat, service, false alarm, etc.Canonical vocabulary is intentionally anchored toward anticipated NERIS families.
source_release_scopeDerived by EmberlinePer-row scope band used to gate all-services charts.Derived from NFIRS release-year bands; future NERIS scope semantics are not yet confirmed.
lat / lon / coordinate_*Not yet confirmedMap coordinates and precision/source provenance.NFIRS geocoded source decision is still blocked; NERIS coordinate availability is unknown.
outcome severity fieldsExpected NERIS mapFatalities, injuries, property loss, and acres burned.NFIRS-backed today; expected to remain an orthogonal outcome overlay in NERIS.
module flags / casualtiesNot yet confirmedModule presence and nested/analytics casualty records.NFIRS module-backed today; NERIS child-record shape must be mapped after public release.
Forward Compatibility · As Designed

The adapter seam

NFIRS was retired January 2026. We designed the pipeline so that only one component changes when NERIS data lands — the Mapper. The storage format, serving layer, and frontend queries are unchanged.

EXTRACTORformat-aware.dbf / .txt / ???SWAPPABLE SLOTNFIRS MAPPERcurrent (retiring)NERIS MAPPER2026+ · drop-inLOADERParquet + ZSTDpartition + sortSERVINGR2 + DuckDB-WASMunchanged↑ only this changes
Drop-in replacement

The CanonicalIncident schema stays fixed. Both NfirsMapper and NerisMapper produce the same output shape. The Loader, R2 storage layout, and DuckDB-WASM serving layer see no difference.

Emberline can visualize NFIRS and NERIS data side-by-side across the 2026 transition boundary — both normalized into the same canonical schema.