We ran one application on seven databases. The benchmark found a bug in our own code.

Database benchmarks are usually very precise measurements of something nobody does. Generate ten million uniform rows, pick a handful of flattering queries, call the driver directly, publish a number with an X after it.

The number is often true. It is also close to useless, because your application does not talk to a driver. It talks through an ORM, an abstraction layer, an access check, a pagination helper and a serializer, and every one of those costs something.

So we asked a different question. What happens when you replace the database underneath a working system and keep everything else — the data, the queries, the indexes, the code path?

We ran it on Franska Bukten, a ventilation inspection system we build and operate. Then we did it seven times: MongoDB, PostgreSQL, CockroachDB, ScyllaDB, Cassandra, CouchDB and FerretDB.

PostgreSQL won seven of the eight read scenarios. MongoDB won the eighth and won the write throughput outright. Everything else lost, and two of them lost by margins that change what you could promise a client.

But the result worth the whole exercise was not a timing. Comparing seven engines on the same data exposed a correctness bug in our own PostgreSQL adapter — one that had been shipping, returning wrong rows, invisible to every test we had.

A benchmark that only produces a ranking has wasted most of its value.

The benchmark does not call the database driver. It calls the application, and the application decides how to reach storage. A read goes through the data service, the query handler, the adapter loader, the data layer and only then the engine.

That means every number includes the real cost of query translation, adapter behaviour, driver behaviour, serialization, the access check, the soft-delete filter, the account filter, alias normalization and pagination.

You could call that unfair to the database. We call it Tuesday. A migration does not happen in a benchmark harness, it happens inside an application, and the application is where the cost shows up.

The setup, stated plainly, including the parts that favour one engine over another:

That is worth holding onto, because wire compatibility is the part of a migration everybody quotes and the part that costs least. Cassandra and ScyllaDB share every line of our CQL adapter, and they still differ by a factor of three on scans. CockroachDB shares every line of our PostgreSQL adapter, and it writes 21 times slower.

The driver was never the hard part. The query model is.

One scenario asks for measurements above a threshold: a range filter on an unindexed number. Five engines returned 165 rows. PostgreSQL returned 210. CockroachDB returned 207.

Row-count parity across engines is the check that makes any timing comparable, and it is the reason this was caught at all. A fast query returning the wrong rows is not an optimization, it is an efficient defect — and a single-engine test has nothing to compare against.

Every numeric range filter on the PostgreSQL engine had been returning wrong rows. Greater-than, less-than, between, and their negations. Any dp system running on that engine had been getting confidently wrong answers, and the wrongness scaled with how many single-digit values the data contained.

The fix casts inside a guard rather than casting blindly, because a schemaless column can hold a non-numeric string and a bare cast aborts the whole query on the first one. A guarded cast yields null instead, which fails to match — exactly what every other engine does with a mistyped field.

After the fix: 165 rows on all seven engines.

Eight scenarios is not a leaderboard and the margins are not one kind of fact. A 1.18-times gap on a deep page and a 21-times gap on writes describe different situations, and averaging them would produce a headline that describes neither.

What the spread tells you is which queries fit an engine and which ones fight it.

PostgreSQL and CockroachDB were not running unindexed on the scan scenarios, and we are not going to present them as if they were.

Our PostgreSQL adapter creates a GIN index over the entire JSONB document, alongside the six parity indexes. So where the other five engines are genuinely scanning, those two are index-assisted. Their 35 ms and 39 ms against MongoDB's 72 ms is not a like-for-like number.

We could have dropped that index for the run. We did not, because it is what dp actually creates on PostgreSQL, and a benchmark of a configuration nobody runs answers a question nobody asked. Declaring the asymmetry is the honest option; hiding it would have been the flattering one.

ScyllaDB is a C++ reimplementation of Cassandra with a shard-per-core architecture, and it is usually the faster of the two. Here it lost every scenario.

This is the most controlled comparison in the whole benchmark. The two share every line of the CQL adapter, the same query planner, the same in-application filtering, the same read-modify-write update path, the same six indexes, the same 250 000 rows. Nothing above the storage engine differs.

So the gap is the runtime and the node's state, and nothing else. Cassandra was sitting at 3.686 GB of its 4 GB heap cap during the run, under real garbage-collection pressure, and still won. We do not have an explanation for that, and an unexplained result stays unexplained rather than getting a story attached to it.

Page 200 with 50 rows: 4.6 ms on PostgreSQL, 5.45 ms on MongoDB, 11 ms on CockroachDB. Then 1.57 s on Cassandra, 5.15 s on ScyllaDB, 30 s on CouchDB.

CQL has no OFFSET. There is no cheap way to skip forward to row ten thousand, so reaching that page means scanning the token range and slicing. A SQL engine walks an index and stops.

A production CQL implementation would not do this. It would use paging-state tokens, cursor pagination, clustering-key continuation or precomputed page boundaries — all of which work well, and none of which give you arbitrary numbered pages.

That is the actual finding. If your API promises page 200, moving to a CQL engine does not change a query. It changes what you can promise the client.

Every other engine in this set degrades when you ask for something awkward. CouchDB refuses. A Mango query that sorts on a field with no index is rejected outright rather than answered slowly, which meant the adapter had to detect that case and sort in the application instead of failing a caller.

It also would not store our documents at all. dp prefixes every foreign key with an underscore — _besiktning_id, _system_id, _ventil_id — and CouchDB reserves that prefix for its own members. The first write came back as a validation error naming the field.

So the adapter renames every field on the way in and back on the way out, in the selectors and the index declarations too. Not a performance characteristic. The one engine here where dp documents are illegal.

What it gets in exchange is the fastest exact count in the set by architecture — one database per collection makes a whole-collection total a metadata read. We measured 29.8 s instead, because our own count implementation was reading a figure that includes CouchDB's index design documents. That was our bug too.

FerretDB presents the MongoDB wire protocol over PostgreSQL storage, which makes it the only candidate in the set that needs no adapter work whatsoever. Our queries arrive unchanged.

Its numbers are the worst here by a wide margin, and one of them is incoherent: indexed lookups measured slower than full scans. A relation lookup on an indexed foreign key took 16.7 s while an unindexed scan of the same collection took 2.29 s. Index access costing seven times a scan is not a performance profile, it is a sign the indexes were accepted and then not used.

We created them through the MongoDB index commands, the same way we did for MongoDB itself, and FerretDB acknowledged them. Whether they reached PostgreSQL as usable indexes, we have not verified.

So the FerretDB column is published as measured and flagged as suspect. The next run checks it with query plans before anybody draws a conclusion from it.

Nothing in this data justifies moving Franska Bukten off MongoDB for performance. PostgreSQL winning most read scenarios is real, and it is not the same thing as a migration being worth it — the wins are milliseconds on queries that were already fast, and the cost is a schema migration plus a GIN index doing quiet work.

Database selection follows workload shape. It does not follow brand reputation, and it certainly does not follow benchmark charts built by whoever is selling the database.

The goal is not to crown a permanent winner. Nobody needs another ranking of databases in the abstract.

The goal is to answer the only question a team actually has: given the system we already run, what would really happen if we migrated it?

This article is a snapshot. The measurements have their own page, which is where new engines land as they get measured: pick any two, switch the factor between median and p95, narrow it to the query shapes your own application performs.

Every invalid run is on that page too, in full, labelled as invalid. That is the part most benchmark pages leave out, and it is the part that decides whether you should believe the rest.