Skip to content
root@payman:~/writing/the-librarian-who-stopped-running$
All posts
12 min read

The Librarian

Why the future of working with documents looks less like search and more like a well kept library, and three things I would do differently.

Moonlit Library of Floating Pages

There is a particular silence that settles over a data room late in the evening. Four hundred documents, grid connection offers, land registry extracts, environmental surveys, a lease with seven amendments, sit in their folders like guests who have long since stopped expecting conversation. And then someone asks a question. Is the substation capacity firm or interruptible? Which of the amendments changed the termination clause? Did the seller ever answer the question about the groundwater?

Every one of those questions has been asked before, in a slightly different form, by a lawyer, a lender or an engineer. And every time, whoever answers it walks back into the archive and begins again.

I spend a good part of my working life in rooms like that. Due diligence on data center sites is, at its heart, the art of asking a fixed body of documents an endless variety of questions. The same is true, I have come to think, of a public authority living among its ordinances and decrees, and of any company whose policies and guidelines have grown, layer upon layer, into a small literature of their own. The documents change slowly. The questions never stop.

The clerk who forgets

Most of what we call enterprise AI today answers these questions in the manner of a diligent but amnesiac clerk. A question arrives, the clerk hurries to the stacks, pulls whatever looks relevant, reads it under the lamp and composes an answer. Then he forgets everything he has learned and returns to his desk to wait for the next visitor.

This is retrieval augmented generation, and it is a fine invention. It made language models useful in places where their own memory was out of date or simply ignorant. But it carries a quiet extravagance in its design. Each question pays the full price of understanding the archive anew. The relationships between documents, the fact that the third amendment overrides a clause in the original lease, that two surveys disagree about the depth of the water table, are rediscovered at every visit and discarded at every departure. Precomputed structures such as knowledge graphs try to remedy this, but they tend to impose a single rigid notion of what matters, usually a catalogue of named entities, and the world’s questions rarely arrive in the shape of the catalogue.

RETRIEVAL, EVERY TIME Q1 A1 Q2 A2 Q3 A3 every question pays the full price again A CURATED LIBRARY archive once catalogue leases grid permits A1 A2 A3 provenance pay once, then ask as often as you like
On the left, every question makes the full trip through the archive. On the right, the archive is curated once into a library that serves many questions and keeps a thread back to its sources.
## A better idea, honestly measured

A recent paper from Stanford and Carnegie Mellon, Librarian Agents, proposes something much closer to how a good human librarian works. Before any question is asked, an agent reads the corpus and builds a workspace from it, an ordinary filesystem of synthesized notes, indices and summaries arranged in a hierarchy, each note carrying links back to the original documents it was drawn from. A second agent, the one who actually answers, then walks through this workspace as one walks through a well ordered library, starting at the catalogue, descending into the right section and reaching for the original only when the summary falls short.

The authors train the builder with reinforcement learning, rewarding it whenever its library makes the answering agent right more often. The results are striking. On a demanding research benchmark of fifty held out tasks, a small sample worth keeping in mind, the system answers roughly three questions in four correctly, where the best conventional retrieval manages fewer than one in two, and on a second benchmark it has never seen, the trained builder still leads, by a wide margin in some domains and by little more than noise in others.

I find the central idea entirely convincing, though not quite for the reason one might first assume. It is tempting to read it as a trade of time, effort spent once at leisure so that every later question becomes cheap. The numbers tell a subtler story. Building one library cost the authors around 180,000 tokens, and every question afterwards still needed about 9,000, because a reader handed a richer library reads more of it. A reader handed a meagre one reads perhaps 1,200 tokens and is wrong far more often. Measured against the untrained builder rather than against conventional retrieval, the authors’ own accounting shows the trained library to be the cheaper way to a correct answer at ten questions per corpus, and, by their projection, the more expensive one somewhere between twenty and fifty. What the curated library buys, then, is not thrift. It is accuracy. In a data room, an audit or a regulatory filing, that is the better bargain by far, since a wrong answer there costs more than any number of tokens.

What the paper does not quite say

And yet the most instructive table in the paper sits in its appendix, and it quietly complicates the headline.

The authors took the finished libraries and subjected them to two small acts of vandalism. In one experiment they flattened the hierarchy, tipping every synthesized note out of its section and laying them all side by side in a single folder. In the other they shuffled the notes, putting each one on the wrong shelf. Then they let the same answering agent loose on the same 435 questions. A word of warning before the numbers. This is a separate run with its own set of questions, so its accuracies and token counts differ from the headline figures, and only the rows of this one table should be compared with each other. This is what they found, in the appendix as Table 7.

Library What was done to it Correct answers Tokens read per question
Original Left exactly as the trained builder built it 55.9 % ± 2.6 3,635
Flattened Every section removed, all notes in one folder 57.7 % ± 2.8 5,497
Shuffled Notes moved into the wrong sections 51.7 % ± 2.9 3,612

Read the first two rows together. Remove every shelf and every section, and accuracy does not fall. It even rises a little, well within the margin of error. The only price is that the reader has to leaf through about half as much text again before it finds its way. The hierarchy, in other words, is a convenience for navigation. It is not where the correct answers come from.

Anyone who has inherited a colleague’s shared drive knows this feeling. Take away every folder and drop all the files into one heap, and you can still find the contract you need. It simply takes longer, because you have to look at more of them. What you cannot recover by searching harder is a contract that was never filed at all. The machine reader in this experiment behaves exactly like that colleague. Missing order slows it down. Missing content defeats it.

The third row tells the same story from the other side. Put the notes on the wrong shelves, so that the catalogue now actively misleads the reader, and accuracy drops by only four points, not far beyond the statistical noise. If the arrangement of shelves were the heart of the method, a lying catalogue would have been a catastrophe. It was an inconvenience.

In fairness, the builder was never rewarded for its architecture. Its training paid only for correct answers, not for tidy shelves or short paths through them, so it is hardly a scandal that it did not learn a meaningful taxonomy. The authors say as much in their appendix. But the main text speaks of reinforcement learning as essential for teaching the builder to structure its memories, and on that point the table counsels modesty. The arrangement of shelves matters far less than that framing suggests. What does matter is the content of the notes themselves, the careful distillation of scattered facts into readable form, tied by a thread to their sources. The appendix hints at this too. On the same document sets the untrained builder leaves behind about three synthesized notes, the trained one more than forty. It learned diligence more than it learned taxonomy.

I would love to conclude that the lesson is liberating, that any capable model told to build a reusable library will get you most of the way without a training pipeline. The paper does not let me. Its untrained builder, given exactly that instruction, produced libraries that answered barely more than a quarter of the questions, worse than plain retrieval. In fairness that builder is a small model of four billion parameters, and it does not even write the notes. It only plans them, deciding what to create, merge or discard, while a larger model with thirty five billion parameters does the writing in both the trained and the untrained version. Training, in other words, changed nothing about how well the notes are written. It changed what was built and how much of it, which is precisely the diligence the table points to. Whether a far larger planner would show that diligence without being taught is a question nobody has yet tested. Until someone does, the honest reading is this. Diligence is what wins, and a small model has to be taught it. A larger one may simply have it, but that remains a hope, not a finding.

None of this diminishes the work, which is careful and honest enough to publish the very table that complicates its story. The idea is better than conventional retrieval. It is simply better in a different way than a single headline number suggests.

The next two chapters, on how I would build such a library and on where it truly earns its keep, are kept under lock and key. A good librarian, after all, does not hand everything to the first visitor. The key is printed on this very page. It is the date on which this essay was published, written as day, month and year, each separated by a full stop, with two digits for the day, two for the month and four for the year. The fourteenth of March 2026, for instance, would be 14.03.2026.

More posts

All posts →
  • A wide field of faint dots narrowing into a single bright point of orange light
    10 min read

    Vertigo

    Most people meet AI as a clumsy chatbot while a few watch the curve go vertical. A view from the narrow end of the funnel, on fear, Europe, security and leadership.