All posts · · 6 min read
Inference that never leaves your machine
For some workloads the interesting question is not which model is best, but whether the data is allowed to leave at all. A model inside the process answers that with no egress, no vendor in the trust chain and no bill per token — and the price it charges is in latency and model size.
Every AI feature starts with a question about quality — which model writes the better summary, which one routes the ticket correctly. Some of them run into a second question that outranks it: is this data allowed to leave the building? A hospital's notes, a bank's transaction memos, a law firm's discovery set, the contents of a customer's S3 bucket that they let you process but not forward. When the answer is no, the model has to come to the data.
Since engine v0.6.27 that is a configuration, not a project. A Synsema process can run a quantized model inside itself — no server next to it, no API key, no socket — and the same program that talks to Anthropic or OpenAI in staging runs against a file on disk in the deployment that has no egress at all.
What changes for the customer conversation§
Three things stop being promises and become properties of the deployment.
There is no egress to allow. The provider is not "a local server on port 11434" that a firewall rule has to trust; it is the same process. A workload can be deployed with network access denied outright and the model still answers. Nothing to allowlist, nothing to log, nothing for a reviewer to take on faith.
There is no vendor in the trust chain. No terms of service to read about training on your inputs, no data processing addendum to negotiate for that feature, no second subprocessor to disclose in the questionnaire your customer sends. The weights are a file you chose, and they sit where you put them.
There is no bill per token. The cost moves to CPU and RAM you are already paying for, which changes which decisions are worth automating: the ones that were not worth a per-call price now are.
Inside an enclave, this is the difference between a demo and a deployment§
If the workload runs in a TEE — the subject of Where to run a confidential workload — then calling an external model API punches a hole straight through the point of the enclave. The data is decrypted inside, and then sent out in a prompt. Whatever the provider does with it, your customer's attestation no longer covers it.
With the model in the process, the prompt never crosses the boundary. That is the version of confidential AI that survives a security review, and it is worth being precise about what it costs: the weights go inside the measured environment, which cuts against keeping the enclave small. The mitigating fact is that weights are data, not code — they do not add a path an attacker can execute — and they are measurable: the engine reports the sha256 of the weights, of the architecture definition and of the engine that ran them, so "which model produced this output" has a byte-exact answer rather than a model name.
What it costs, stated plainly§
This is CPU inference, and the numbers are the ones you would expect from CPU inference. Prefill is around 12 tokens per second, so a 1,000-token prompt takes about 90 seconds on a small model. Generation runs at roughly 11 tokens per second on a 0.5B model and 5 on a 3B with four threads. Loading the model takes 7 seconds for a 0.5B and about 35 for a 3B — paid once per process, so under a served deployment the first request pays it and the rest do not (measured: 8.2 s down to 1.3 s).
That profile has a clear shape. Classification, routing, extraction, redaction, a short rewrite, a summary of a few paragraphs: right tool. A chat assistant answering 4,000-token prompts with long replies on a CPU box: wrong tool, and no configuration will change that. The honest architecture for most products is both — the small local model for the high-volume decisions on data that cannot move, a frontier model over the network for the work where quality is the constraint and the input is not sensitive.
Nothing gets downloaded, which matters more than it sounds§
The model is named the way your machine already names it: a path to a .gguf, a model:tag that is already in the Ollama cache, or a Hugging Face repo that is already in that cache. None of the three forms fetches a byte. A production process that starts by downloading 3 GB from a host it does not control is a supply chain event and an outage waiting for a bad network day; here the weights are provisioned like any other artifact, and the process only reads what is there.
The engine also stopped needing a release from us to support a new model family. An architecture is a text file the compiled binary reads at startup — a flat list of named steps over the tensors of the file — so a model that the engine does not know can be added by whoever deploys it, without a compiler and without waiting. The file format has no conditionals, no loops and no way to open a file or a socket, so accepting a definition from outside is not accepting code. The developer-side detail is in A model you already have.
The same release did it for decisions§
The other half of this is the judge: a System One model that returns calibrated probabilities instead of prose, which is what most automation actually needs. That now runs locally too, from a checkpoint on disk with no key and no network — including the confidence gate that decides what a person sees. That is A judge with no vendor.
Where to start§
Take the one feature that is blocked on a data question rather than a quality question — the one where somebody in the room said "we can't send that" and the feature went back in the drawer. Check whether its model work is classification or short generation. If it is, it fits in the process today: one variable names the provider, one names a model your machine already has, and the deployment it runs in needs no egress at all.