Pipeline Ops

Every ingest run writes exactly one cvm_ingest_log row, so this page is the audit trail — and the answer to "has anything quietly stopped landing". Read it before quoting any number on this site as current: a page rendering perfectly from three-month-old data looks exactly like one rendering from today's.

The failure this page catches is not the loud one. An erroring run is visible everywhere; a cron that never fires logs nothing at all, and a slice reporting ok while its table stops advancing looks healthy in the log. So the log and the tables are both shown — a disagreement between them is the thing to look for.

How to read the health strip. The cron runs daily at 06:00 UTC.

• Hours Since Last Run above ~30 — the schedule itself has stopped. The silent outage: a pipeline that never runs logs no errors at all.
• Stuck 'running' — began over six hours ago, never wrote a terminal status. A process that died mid-run.
• Errors (7d) — includes slices that fetched rows but upserted none.

Rows Ingested per Day

Last 30 days, one bar per calendar day. A day with no bar is a day the pipeline did not run — the day spine keeps empty days on the axis instead of dropping them, which is precisely what makes an outage visible rather than invisible.

The same 30 days by outcome below. skipped is expected and healthy — it is how a month CVM has not published yet is recorded. error and running are not.

Freshness — Last Successful Slice per Entity

Sorted worst-first. Days Since OK counts only successful runs, because a 404 (skipped) and a failure (error) both mean no data arrived.

The tell for a silent outage is the pair of columns on the right: a stale Last OK next to a recent Last Attempt whose status is not ok. An entity that has never succeeded sorts to the very top with a blank date.

Status Breakdown per Dataset

Every (entity, doc_type) seen in the last 30 days, worst-first by error count, with the most recent error text. An entity that produced no runs at all still appears — as a row with blank counts, which is the loudest signal on the page.

Entity and doc-type labels here are the ones the pipeline writes to the log, not the old Flask dispatch keys: the log says inf_diario where that matrix called the same dataset diario.

Table Freshness — What Actually Landed

Read from the ingest tables themselves, not from the log. Each table is dated on its own natural key column, named in Date Column so grains are never silently mixed — cvm_fip_periodic is yearly and is shown at 31-Dec of its newest reporting year purely so the column is comparable.

Expect a structural lag: CVM publishes monthly datasets 1–2 months in arrears, so a Days Stale of 30–90 on a monthly table is normal. An empty table sorts to the top with a blank date.

Newest Row and Days Stale are exact. Rows (≈) is a planner estimate (pg_class.reltuples, refreshed by the ANALYZE the daily ingest runs after each upsert — within ~1% of truth); a 0 means the table has never been analyzed. Exact counts here cost a full scan of every table on every build.

Analytical Coverage per Month

The downstream half: distinct funds present per month in fact_fund_monthly, by family. The log can say ok while the analytical layer still has a hole, and a month whose fund count collapses is a data problem the log alone would not show.

Structural gaps that are not failures: FIP reports yearly (only its December month is populated), FIAGRO's monthly file only begins 2025-05, and the newest month or two are thin because of CVM's publication lag. Every page that depends on these families states the same caveat where it bites — see Industry Structure and Performance.

Most Recent Runs

The 20 newest rows of cvm_ingest_log, newest first. Duration is blank while a run is still running — the tell for a job that died without writing its terminal status. Blank rows at the bottom mean the log holds fewer than 20 runs.

Triage runbook for what to do with a red row: docs/reference/DATABASE_MAINTENANCE.md.