Search, discovery and ranking
Why is usage-based ranking a feedback loop, and how do you damp it?
Discovery is an information-retrieval problem. It reuses ordinary text search: an inverted index (a map from each word to the assets that contain it) over names, descriptions and column names, with a scoring formula such as BM25 (a standard score that rewards rare matching words and penalizes very long documents). That part works and is not interesting.
What does not work is relevance from names alone, because asset names are short, non-descriptive and heavily duplicated. The signals that work come from usage: query counts, number of distinct users, number of downstream edges, recency. The best ranking is PageRank-shaped: an asset is important if important consumers depend on it. (PageRank is the algorithm that ranks web pages by who links to them. Here, "links" are lineage edges and reads.)
The real problem is not finding assets. It is choosing among forty tables named something like "revenue". The needed fact is which one is canonical. That fact is either:
- asserted through certification (a person approves it; it decays), or
- observed through concentration of usage (it does not decay, but it has a nastier property).
Usage ranking is a feedback loop
The top-ranked table gets used more, which raises its rank, which gets it used more. This is a positive feedback loop ("rich get richer"). It locks in whichever table happened to be popular early, including a deprecated one.
The share keeps rising, whether or not A is the right table.
The escape is the same as in any positive feedback system: a damping term injected on purpose. Usually this is certification, or weighting by recency, so that a table's early popularity cannot dominate forever. And note the loop is invisible because nothing fails: the system just converges on the wrong canonical table.
Usage ranking is the only non-decaying relevance signal, and it is a positive feedback loop that will happily canonicalize the wrong table.