jwogrady Working notes on tools, and why they earn their place

The database that held no data

resolves JRN 26-041

Twenty-eight views, zero tables. It was not an archive, it was a set of directions.

Before you move a business’s data anywhere, you take an inventory. It is dull work and it is the only part that is genuinely load-bearing, because everything after it is built on believing you know what you have.

I had a file called wtp.duckdb. Two hundred and seventy-four kilobytes, sat in the archive directory next to everything else, named after the company. If you had asked me what it was I would have said: that’s the database, that’s the historical record, that’s the thing we must not lose.

It contains no data whatsoever.

A terminal card. The file listing shows wtp.duckdb at 274432 bytes. A catalog query returns one row: VIEW, 28 — no tables at all. Counting rows from inside the collect directory fails with an IO Error about files not found. The same query from the parent directory returns 475.
The same file, the same query, two working directories. One of them is an archive and the other is an error message.Screenshot

Twenty-eight objects in it, and every single one is a view — a saved question rather than a stored answer. SELECT written down and given a name. A view holds no rows of its own; it goes and fetches them from somewhere else at the moment you ask, and the somewhere else here was eighty-two JSON files sitting in a sibling folder, eleven and a half megabytes of them, which is where the actual record lived the entire time.

So that 274 KB is not a small database. It is not a database. It is a quarter of a megabyte of pointing.

And here is the bit that took it from a curiosity to something I am going to remember. Look at the middle command on that card. That is the same file, opened with the same tool, running the same query — and it fails. The only difference is which directory I was standing in when I ran it.

The views resolve their files by relative path. Stand one folder too deep and ga4_daily is not a table with 475 rows in it, it is an IO Error about a pattern that matches nothing. Step back up one level and the 475 rows come back.

Twenty-eight views, zero tables. It was not an archive, it was a set of directions.

Directions are enormously useful and I do not want to be unfair to whoever built it — querying JSON files in place, without importing them first, is a genuinely good technique and the reason the thing is 274 KB instead of twelve megabytes. The failure is not in the design. It is that a file with .duckdb on the end looks like a container, and I would have treated it as one. Copy that one file to a new machine, feel like you have rescued the archive, delete the folder full of JSON because it is bulky and looks like scratch output — and you have kept the index of a book you have thrown away.

The same afternoon handed me the smaller version of the same lesson, which is what convinced me it was a pattern and not a quirk.

The archive has four folders in it, one per source the data was pulled from. Four vendors, four captures, tidy. One of those folders contains zero files. The capture failed — a permissions refusal from the vendor’s API — and it failed at the very start, so there was nothing to write.

Now, the tool that did the capture behaved impeccably. It wrote a manifest that says complete: false and records the reason in plain text. The honest account was right there. But nothing about the folder says any of that. It has the vendor’s name on it, it sits in a row with three folders that worked, and at a glance — which is the only kind of look an inventory gets — four sources looked like four sources.

A directory named after a thing is not evidence you have the thing.

What I am taking out of this, and it has nothing to do with databases: the unit you are storing is almost never the file you think it is. A spreadsheet whose numbers come from links to three other workbooks. A slide deck with a chart that lives in someone else’s file. A contract that references a schedule of rates attached separately, which is where all the actual money is. Each one is a view. Each one is fine, right up to the moment it is moved, or the folder beside it is tidied, or the person who knew the layout leaves.

The cheap test, and it took me one command today: pick it up and carry it somewhere else, and see if it still answers. Not open it where it lives, where everything it depends on is quietly within reach. Move it first. Break it on purpose, now, while it is still only an experiment and not a recovery.

Two hundred and seventy-four kilobytes of pointing. Move it alone and you have moved nothing at all.