The state is objects.
Everything else follows.
Data Oil's engine is written for object storage rather than for disk. No local disk to lose, no volume to resize, no leader election to reason about - and a node that dies takes nothing with it.
Design decisions
Seven decisions, and what each buys you
The decision that matters most
Every durable byte lives in the object store, and the writer lease lives there too. One writer per database, as in PostgreSQL; the difference is that the lease is arbitrated by the store, not by a process that can lose contact with its replica. Two servers can never both believe they are the writer, and a replacement takes over from the store, so a node that dies takes nothing with it.
What that removes from your design
No local disk to lose. No volume to resize. No leader election. No split brain to plan for. Moving cloud is a connection-string change, and one tenancy may hold databases on several providers at once.
+All seven decisions and their consequencesOpen the seven
| Decision | Consequence |
|---|---|
| Every durable byte lives in the object store | Compute is replaceable; storage is the durable part, and it is the cheapest durable byte anybody sells. |
| One writer per database, as in PostgreSQL, with the lease in the store | No split brain, by construction: the store arbitrates, so two servers can never both hold the lease. A replacement takes over from the store, and the promotion is explicit and auditable. |
| Snapshot isolation, optimistic commit | Readers never block writers; a conflicting commit is a 409 with X-Retryable: true, so a client knows what to do without parsing anything. |
| Change feed off the write-ahead log | A transaction's changes are emitted contiguously after its commit; an aborted transaction is never published. Resume from a bookmark over a WebSocket. |
| Multi-tenancy inside the engine | A tenant's quota refuses in the middleware before a statement is parsed, and the refusal carries used, allowed and resetsAt. |
| Provider-neutral store | s3, oci, azureblob, Google's interoperability endpoint and MinIO behind one connection string. Moving cloud is a string change, and one customer may hold databases on several at once. |
| Placement per database | A database sits on a named instance with a region; residency is enforced per write against that region, and an outbound model endpoint is assigned per database. |
One engine, four index kinds
One planner over one storage engine, inside one transaction
Ordered, unique, full-text (BM25) and vector (HNSW) indexes, written from scratch for this engine, over document, vertex and edge collections. A write is visible to all four the moment it commits.
SELECT title,
VECTOR_DISTANCE(embedding, EMBED('casing failure'), 'cosine') AS distance,
$score AS keywordScore
FROM Report
WHERE SEARCH_INDEX('Report[body]', 'casing failure') = true
AND filedAt > date('2026-01-01')
ORDER BY distance
LIMIT 10;
+The SQL surface, enrichment rules, descriptions and shardingOpen the detail
The SQL surface
Joins, window functions, common table expressions, set operations, graph traversal and pattern matching, full-text and vector functions, and scripting for several statements in one transaction. EXPLAIN gives the plan and the access path; PROFILE runs it and reports what each operator actually did.
Enrichment as standing rules
Chunking, embedding, entity extraction, transcription and translation into eighteen languages are declared once as rules, and they keep up with every change.
Descriptions are schema
A description on a type or a property is carried into the schema endpoints and into /schema/narrative, the catalogue the natural-language layer plans against. A description is data, never an instruction.
Sharded when you need it
Shard maps and key routing across nodes, promote and step-down, and a console that brings a new node up on a machine you name.
Deployment
Deploy it where the data has to be
Hosted by Data Oil
On infrastructure we run, over object storage, in the region you name. One bill, one rate card, and no provider's free tier priced into it.
In your own cloud account
Your buckets, your keys, your compliance perimeter. The console brings a new node up on a machine you name, end to end, and keeps the credential nowhere.
+Several regions at once, shared or dedicated, your AI servers, and four ways inOpen
Several at once
One database on Oracle Cloud in Frankfurt, one on Azure in Melbourne, under one tenancy, with residency enforced per write against each instance's region and a model endpoint assigned per database.
Shared or dedicated
A node of your own at a fixed size, or a share of one, charged minute by minute for your share at $0.060 a processor-hour and $0.008 a GiB-hour. A shared node grows with demand and comes back down when it falls, within its size’s floor and ceiling; work in flight drains first, and writes pause for seconds, refused safely rather than lost. Your charges name the size and the hours.
AI servers you allocate
An administrator attaches the servers that do translation, embedding, transcription and the assistant, through the UI or the API. Credentials are sealed and shown only by fingerprint; work is spread across the servers and moves on when one stops answering. More translation at once is more servers, each with its own GPU.
Four ways in
HTTP /api/v2, embedded in a .NET process with no socket, the PostgreSQL wire protocol for any driver or BI tool, and a command line. The same engine and the same statements on all four.
Security
Secured in layers, and every layer checkable
Identity, authorisation, keys, network, secrets, residency, audit and the model boundary: eight layers, each with a refusal you can provoke and an audit line you can read.
+The eight layersOpen the layers
| Layer | What there is |
|---|---|
| Identity | OIDC with PKCE, RS256 only, issuer and audience checked. A directory group with no mapping grants nothing; there is deliberately no default role, and a mapping cannot grant administrator. |
| Authorisation | Roles grant the four verbs per collection and optionally per property. A column somebody may not read is dropped from SELECT *; naming it is refused; a column classified read-only is read-only for everybody. Both query surfaces enforce it. |
| Keys | An API key can be named, scoped to databases and verbs, listed, refreshed with the old one still working while callers roll over, and revoked. A key may not mint a key. Introspection answers active:false identically for unknown, expired and revoked. |
| Network | Address allowlisting per customer, in CIDR, writable by the customer's own administrator. A machine plane on a second port that customers cannot reach. |
| Secrets | Sealed at rest; a model key or an AI server’s credential is never returned once registered. |
| Residency | Enforced per write against the region of the instance the database sits on. A write outside the declared region is refused and nothing is stored. |
| Audit | Every statement with its shape and never its values, every administrative act, and every act on the console: a page opened, a control pressed, a sign-in. Nothing on the customer surface edits or removes a line. |
| The model boundary | The engine loads no model for the final answer. POST /query/prepare runs every stage inside the deployment and hands back the exact submission, prompt, evidence and byte count, without sending it. |
Operations
Written so people who did not build it can run it
Built so that "nothing happened" and "the check never ran" can never look the same to an operator. 534 named reliability guarantees run on every build of the engine.
+What an operator has at three in the morningOpen the table
| You need to | What there is |
|---|---|
| Know what is wrong at three in the morning | dataoil doctor: findings worst first, every one with a next step, exit code 2 when anything is critical. A check that could not run is a finding, not a silence. |
| Trust a backup | Backups are chained incrementals, proved by a real restore drill into a scratch name on the cadence you chose. The console shows proved, stale or never per database. |
| Restore without an accident | A customer restores their own backup into a new name from the console; a restore over a live database must name the database in confirm, and an unconfirmed attempt changes nothing and is written to the audit. |
| Know how far behind a database is | Four independent measures, log bytes, log objects, memtables waiting, bytes above the level targets, on the console, because a database that is fine on three of them is not fine. |
| Know a schema drifted | dataoil schema diff a b exits 2 when they differ, so a pipeline can gate on it. |
| Get space back | The reclaim survey separates space a deletion returns at once from space that needs segments rewritten, and the number offered is the number taken. |
Prove the architecture
A four-week proof of concept, designed so that it can fail
Week one it deploys where your data has to be, against your object store, with SSO against your directory. Week two your real data, in your real shapes, indexed four ways. Week three the query you cannot run today, measured against the one you run now. Week four your model, your key and exactly what would leave. Week five a backup drill you watched. Week six the written answer.
Questions this page raises
Three for your next design review
Who owns the job that keeps the search index in step with the rows?
And what happens to the index when that job is behind?
Read the answer →How does your current store decide who the writer is?
A lease in the object store is a decision two servers cannot both make.
Read the answer →What stops a write landing in the wrong region?
A policy document, or a refusal?
Read the answer →