Most ML architectures assume they will be rebuilt in eighteen months. The systems we design for industry and the public sector don't get that luxury: a quality-inspection line or a case-handling system has to keep running when the vendor that supplied a component loses interest, changes its pricing, or disappears. So we start from a different question. Not "what is the best tool today?" but "what does it cost to replace each part when we have to?"
This note describes the layered stack we reach for when the answer to that question matters for a decade. It's a reference, not a mandate. Every layer has alternatives, and that's exactly the point.
Start where the data lives
The data plane is the layer we change least. Production data usually already sits in systems that work: a process historian, an MES database, a records archive. We build versioned ingestion and feature pipelines against those systems rather than migrating everything into a new platform first. Migration projects are where ML initiatives go to stall, and every copy of the data is another surface to secure and another lineage question to answer.
The pipelines themselves are code: version-controlled, reviewed, and reproducible. Given the same inputs, a training set can be rebuilt bit-for-bit. That property does more for both debugging and compliance than any dashboard.
Open-weight models, in a registry you control
Every model in the system, whether trained from scratch or fine-tuned from an open-weight base, lives in a registry on the client's own infrastructure. Each entry holds the weights, the training configuration, a reference to the exact data snapshot, and the evaluation results. Nothing sits solely behind someone else's API.
This is the layer where sovereignty is won or lost. If the weights are on your disk, a provider changing terms is an inconvenience. If they aren't, it's an outage you can't fix.
Portable serving
We serve through standard formats and interchangeable runtimes: models exported to ONNX where the architecture allows, served with ONNX Runtime or Triton on Kubernetes, with Ray Serve when a pipeline needs Python-level composition around the model. The specific tools matter less than what they share: open interfaces, open source, and at least one credible replacement each.
We prove the portability instead of assuming it: the export path to a second runtime is exercised in CI, not discovered during a crisis.
The evaluation harness comes first
Before serving, sometimes before the first model, we build the evaluation harness: versioned test sets, agreed metrics, and a runner that executes in CI. It looks like overhead. It's the opposite: the harness is what makes every later replacement safe. Swap the model, the runtime, or the hardware; re-run the harness; compare the numbers. Without it, every upgrade is a leap of faith, which in practice means upgrades stop happening.
Monitoring, drift, and evidence
In production, the same harness logic watches live traffic: input distributions, output distributions, latency, and cost, with drift detection that pages a person before it degrades a decision. And because the pipelines are versioned and the registry records lineage, compliance evidence is a first-class output of the system: model documentation, data lineage, and decision logs are generated by the same machinery that ships the model, not reconstructed by hand months later for an auditor.
The exit-path test
The architecture check we apply at the end is simple. For each layer, ask: what does it cost to replace this? Models: the weights and training code are yours, so the cost is a fine-tune. Serving: the second runtime already passes CI, so the cost is a deployment. Data plane: the data never moved, so the cost is a new pipeline. Orchestration: Kubernetes is an open standard with many vendors behind it.
If the answer at any layer is "a rewrite", the architecture isn't sovereign. It's just self-hosted lock-in.
This isn't the fastest path to a demo. It is, in our experience, the fastest path to a system that's still running, and still yours, in year ten.