How Randomly datasets get from public sources into a small sample

Most “random data” sites either ship a tiny hardcoded list or dump a CSV into your laptop and call it an API. I wanted a third option. Randomly keeps large catalogs on the server, versioned, attributable, and sampled. Your browser gets a handful of rows. It does not get GeoNames. It does not get the SSA baby-name archive. That split is the whole architecture.

This is the tour I give myself when a generator looks empty, when a filter 404s, or when someone asks why we will not email them the full cities file. If you only need a sample, use the tool pages. If you want the contract, the public surface is documented on API info.

Why the browser should not eat the dump

A city file with tens of thousands of rows is fine on a laptop. It is rude as a first paint on a phone. Unicode emoji catalogs, word lists, and name tables add up. If I bundled every catalog into the JavaScript, the site would be a download masquerading as a homepage. I also do not want to republish someone else’s bulk dump as a static asset with my domain on it without a license gate. Sampling through a Worker lets me enforce count caps, filters, and attribution in one place.

There is a fallback. Some generators still carry a tiny bundled snapshot so a Worker hiccup does not show a dead button. That snapshot is not the production catalog. Production traffic is supposed to hit the current published release. If you get repeats that feel “too small,” you might be on the fallback. That is a bug in availability, not a feature of the dataset.

I care about this because people treat whatever they see as the population. If the UI can only pick from 40 cities, users write blog posts that say Randomly has 40 cities. The live catalog is larger. The page should say so, and the API should prove it.

Shards in R2

Published releases live in Cloudflare R2 under a prefix like datasets/<slug>/<version>/. You get a manifest.json plus shard files shards/00000.json, shards/00001.json, and so on. Each shard is a JSON array. We cap a shard at 2,000 records so a read stays bounded. The Worker does not stream the whole prefix. It picks indexes, figures out which shard holds that index, and reads only what it needs.

R2 objects are immutable. We do not overwrite a version. A bad import does not mutate yesterday’s file. It publishes a new version id. That makes rollbacks boring, which is the nicest sentence in operations. If a license review fails after upload, the version never becomes current. Orphans can exist in the bucket. Current is a D1 pointer, not “whatever file was written last.”

The manifest stores record counts, shard counts, a content SHA-256, source version, license, attribution, timestamps, quality totals, and the filters we allow. Filters are not free-text SQL. If the manifest does not list a field, the API will not filter on it. That keeps a client from probing columns we did not mean to expose.

The D1 version catalog

D1 holds the catalog: which slugs exist, which version is current, shard counts as JSON, license strings, and the boring bookkeeping that lets a request resolve cities without listing the bucket. Publish is staged. A version can sit as something other than current while objects finish uploading. A trigger flips current only when the manifest and every shard made it. That avoids the classic “catalog says 40 shards, bucket has 39” outage.

Superseded versions stay around for rollback. I would rather waste a few gigabytes than reconstruct a GeoNames extract from memory at 2 a.m. When we activate a new version, the old one is marked superseded, not deleted in the same breath.

Unknown slugs return a JSON error that does not dump internal table names. Unpublished datasets should look like unknown datasets to the public API. Curiosity is not a reason to leak a staging slug.

Where the rows come from

Sources are not a mystery blog roll. Cities come from GeoNames bulk files. US given names come from SSA public data (and some Census files for surnames and related tables). Emoji and territory data come from Unicode and CLDR. Words come from Open English WordNet. There are other catalogs: Gazetteer ZIPs, Open Library, Wikidata extracts, USDA foods, MusicBrainz, World English Bible verses, libphonenumber metadata. Live movie and TV picks are a different path: TMDB on demand, not a frozen shard, because that catalog changes too fast to pretend it is a monthly ZIP.

Each source has a license we have to accept. NOASSERTION does not ship. PokéAPI and YouTube stay in legal review until redistribution is actually allowed. I would rather a missing generator than a cute infringement. The registry that names adapters and refresh tiers lives in our data scripts, not in the browser bundle.

Refresh is uneven on purpose. Telephone metadata can move weekly. Cities are more monthly. Unicode rides official releases. Census geography is annual-ish. Heavy dumps (Wikidata, Open Library, MusicBrainz) are manual because they are huge and because “weekly” would be a vanity cron. SSA’s name files are a manual batch for similar reasons: not because names are secret, because the fetch is annoying and should be operator-attended.

What the Worker actually returns

The public dataset API looks like GET /api/v1/datasets, GET /api/v1/datasets/:slug, and GET /api/v1/datasets/:slug/random?count=12. Count is capped (100 on datasets). Filters are capped. Shard reads per request are capped. Seeds exist so a demo can be deterministic. Repeats are off by default so a handful of cities are a handful of cities.

That is sampling, not pagination through the dump. There is no “download all” on the random endpoint. If you need the upstream file, go to GeoNames or SSA and take it under their terms. We are a sampler with attribution, not a mirror.

Live tools (movies, some TV) use /api/v1/tools/:slug/random with a smaller batch cap and a short timeout. Those responses carry source attribution and cache metadata because the origin is not our R2. When TMDB is sad, the UI should say so rather than silently serving a 2019 snapshot and calling it now.

Seeds, filters, and honesty

Deterministic seeds are for tests and for “share this exact draw.” They are not encryption. If you pass seed=demo, you should get a stable draw for that catalog version. Change the version, and the draw can change, because indexes point at a different array. I document that so QA does not file “the seed broke” after a cities refresh.

Filters must match the manifest. A gender filter on baby names makes sense. A gender filter on cities does not. Fail closed. Silent ignore is how you get a UI that claims “female names in France” and returns random names from everywhere.

Quality gates run before publish: normalize, deterministic dedupe, shard, hash. An adapter that emits zero rows cannot become current. That sounds obvious until you parse a ZIP with the wrong filename and upload an empty success.

What this means for the name tools

The baby names page is not scraping SSA on each click. It samples a published slice of SSA popularity data. That data is US-centric. It is not a world census. I wrote the longer caution in baby names. Same family of idea as this post: the browser sees a draw, the archive stays upstream.

American given-name and surname tools follow the same pattern. If a count on the page disagrees with SSA’s website, check our version string before assuming we invented names. We probably have last year’s file, or a filtered subset, or both.

What I will not do

I will not put raw PII catalogs in R2. SSA baby names are aggregates. That is different from a file of people’s SSNs. Format generators for SSN, IBAN, and cards are computed, not sampled from victims. See PII-safe test data and SSN format-only.

I will not expose shard indexes as a browsing UI. If you can walk shard 0, then 1, then 2, you have the dump. The product is random access with a cap, on purpose.

I will not pretend a 12-row response is statistically optimal sampling theory. It is uniform-enough draws over published indexes with a cap on how many shards we touch. Good enough for a city prompt. Not good enough for a paper on urbanization.

How I debug a dead generator

First, is D1 pointing at a current version for that slug? Second, does R2 have the manifest and the shard count the catalog claims? Third, is the Worker hitting the cap on shard reads because a filter is rare? A filter that matches 3 rows in shard 17 will look “broken” if we refuse to scan the world. That is a product choice. Widen the filter or accept a smaller population.

Fourth, am I looking at the fallback snapshot? If the network tab never calls /api/v1/datasets/..., the UI is not in production mode. Fifth, license. A source that flipped terms should fail the gate rather than quietly keep serving.

The API info page is the public version of this. This blog post is the opinionated one: why the dump stays on the server, why versions are immutable, and why your browser is not a replica of GeoNames.