Scenarios

Hourly partitions over three years are 26,280 per table, each a row with its statistics in one relational metastore. Planning reads thousands of rows, so it slows as the table ages. By Little's Law, at 20 plans a second, 2 seconds of planning holds 40 slots, against 10 at half a second.

The small-file problem is a metadata problem first. And the streaming job commits too often and loses compare-and-swap races: with a 100 ms catalog round trip, the ceiling is 10 commits a second.

Daily partitions with larger files (1,100 partitions instead of 26,280); a manifest-based table format, so pruning skips whole manifests; commits every 10 seconds, with retries that re-validate; and metadata compaction.

Data becomes visible up to 10 seconds later, and the manifest format needs its own compaction job, since the small-file problem recurs one level up. Measure planning time before and after.

Identity was the name, so each rename was a delete plus a create: lineage edges were cut, and usage history reset to zero, which is why the tables now look unused.

Pause the bot. Rebuild the mapping from old to new names from the rename log, and reattach the history.

Stable asset IDs, with names as attributes resolved late; key lineage, ownership and usage on the ID. The cost is a layer of indirection and a resolution service that needs its own availability.

"Zero use in 90 days" is a guess about a gap that cannot be observed. Revoke access for 30 days first, logging any denied requests, and only then delete.

Each writer uploads its data files first (invisible, and safe to repeat), writes a new metadata tree, then compare-and-swaps the catalog pointer. A writer that loses re-reads the new state, checks whether the changes conflict (the same files or partitions), and retries or fails.

The catalog must support an atomic conditional update; a plain file in the bucket cannot. One commit per round trip, so the stream commits in micro-batches.

1. Orphan files from a crash between upload and commit: clean them later, with an age threshold longer than the slowest writer (8-hour jobs need over 8 hours, with margin).
2. A catalog outage stops every write: the catalog is the single point of ordering, so make it highly available.
3. Too many small manifests: schedule metadata compaction.

A pointer store that is only eventually consistent cannot give serializable tables.

A full scan of 50 TB at 5 GB/s takes about 2.8 hours, so analyzing every hour is not even possible. And even if it were, statistics would be stale exactly where data changes fastest.

Statistics written into the manifests at commit time (min, max, counts, null counts), plus mergeable sketches for distinct counts, so each new file brings exact numbers with it.

With four or more joins, estimation errors compound: 3× per join becomes 81×. Add adaptive re-planning that uses the row counts actually observed. The optimizer needs alternatives ranked in the right order, not exact numbers.

A small cost on every commit, and approximate sketches. Write-time statistics first, adaptive re-planning second, and no hourly scans.

Lineage can prove a path exists, not that none does: "no path found" means "no path seen". State the coverage, say 70% of jobs emit lineage, and list the blind spots: notebooks, vendor ETL, dynamic SQL, UDFs.

1. Raise coverage, with engine hooks and by routing notebooks through the engine.
2. Deny the export account any read of unlabelled columns by default, logging denials.
3. Scan the export's contents for strings shaped like email addresses.
4. Propagate labels from the source columns.

Labels are best-effort, bounded by lineage's completeness; the control is the denial channel plus content scans, not the graph.

False positives from pattern scans, and a denial channel that blocks legitimate jobs, so roll it out in report-only mode first.

Metadata that is not a by-product of execution decays into noise.

Stable asset IDs, a manifest-based table format, a catalog with atomic compare-and-swap; statistics written at commit time; provenance columns per file or batch; branches for a write–audit–publish flow.

Lineage emitted by engines at run time, plus static parsing for code paths that did not run, reconciled together, with coverage as the headline metric. Labels propagated along lineage, with default deny.

Ownership derived from code repositories and deploy identities and resolved to teams; search ranked by usage from query logs, with damping; telemetry kept separate from audit, which is append-only and hash-chained.

Glossary terms and descriptions only where no source exists, each with a "last confirmed" stamp that fades; back them up, since they cannot be rebuilt.

The catalog is now on the critical path, coverage is limited to instrumented engines, and exchanging metadata between tools gives the lowest common denominator of meaning.

Absence of use cannot be proven, so build up evidence and keep every step reversible.

1. Use lineage to narrow candidates to tables with no declared consumers.
2. Rank candidates by last read from query logs, and flag periodic patterns (monthly, quarterly, yearly). Exempt anything read in the last 400 days, or carrying a regulatory tag.
3. Revoke read access for 30 days, logging denials, with an easy way to ask for it back.
4. Only then delete, metadata first and data second, keeping the data recoverable for a grace period.

A yearly job that fails to log its reads; ownership records too stale to ask anyone, so notify the team resolved from the code repository.

Delay and some storage, against the much higher cost of a wrong delete.

How old a table version readers may see: it is an isolation decision.

Without a cache, each plan costs a catalog round trip. At 5,000 queries a minute (about 83 a second) and 50 ms each, that is about 4 concurrent catalog requests (Little's Law) and 50 ms added per query: tolerable for a healthy catalog, but it becomes a bottleneck once the catalog also serves commits. A 60-second cache cuts catalog reads to a few per table per minute, but readers may see data up to 60 seconds old, and two cached tables may sit at different versions.

A short lifetime, seconds, for tables where freshness is part of the product. Or cache by snapshot ID, which is immutable and so correct forever, and refresh only the pointer on a short lifetime.

State the staleness bound explicitly; do not leave it hidden in a latency setting.

Stale statistics. Compare the planner's estimated rows with actual rows in the plan, and check how old the statistics are.

Refresh the statistics, or force the previous join order.

Maintain algebraic statistics as part of every write; use HyperLogLog sketches, which merge, for distinct counts; refresh holistic statistics on a cost-based trigger; and show the statistics' age in plans. Treat distinct counts estimated from samples as hints: they have proven limits.

That a clustering rewrite did not invalidate an index or delete vectors by moving the rows they point to.