The Best Material Is Not On The Open Web At All

Vast, carefully organised collections exist that a general search will never show you.

There is a persistent assumption that the open web is a reasonable proxy for everything written down. It is not close. Enormous quantities of material sit in collections that are public and free to consult but structurally invisible to a general search: catalogued holdings, deposited records, historical papers, official gazettes, technical collections, local history material, maps and plans, photographic and audio collections, and the accumulated documentation of institutions that have been keeping paperwork for a century or more. Much of it is described in catalogues rather than reproduced in full, so the words you would search for do not exist as text anywhere a general tool can reach.

This changes how you should search when the open web disappoints. Instead of trying harder in the same place, ask which kind of institution would hold this. Records of what an organisation did are held by the organisation or by whoever inherited its papers. Anything to do with land and buildings leaves a paper trail with the bodies that administer land. Local matters live in local collections, which are frequently indexed by people who care enormously and are used by almost nobody. Published material has been systematically collected somewhere by law or by habit.

Then search the catalogue rather than the world. Catalogue searching is a different discipline: the descriptions are terse, written in the conventions of the people who made them, often decades old, and they reward guessing what a cataloguer would have called the thing rather than what you would call it. It is slower to learn than typing into a general box and it repays the effort enormously, because you are searching an organised space rather than an unruly one.