The Signal · Issue 02 · Open Source
An open inference engine that can give the same answer twice
DeepSeek open-sourced DeepInfer, the framework behind its production service, with batch-invariant inference and deterministic sampling. For regulated users that may matter more than the speed figures. So does the US order that arrived the day before.
Facts are attributed and linked to their sources. The analysis and opinions are the author's own.
What it is
On 20 October DeepSeek open-sourced DeepInfer, the inference framework behind its own production services, under the Apache 2.0 licence. The repository supports one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and offers offline inference only. A KV cache system, prefill/decode disaggregation and load balancing across replicas are not included.
Two features stand out. "Batch-invariant inference" keeps a model's results identical regardless of what else is in the batch. "Deterministic sampling" produces identical outputs for identical inputs and settings. Both work only on NVIDIA hardware. SiliconANGLE notes that the published benchmarks are DeepSeek's own, run on NVIDIA chips. According to DeepSeek's deployment notes, as reported by SiliconANGLE, one cluster serves 155 trillion tokens a day, and the smallest efficient deployment uses 64 GPUs for prefill and 64 for decoding.
The release came a day after the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, including third-party services built on its technology (Federal Register notice).
Why it matters
Speed numbers get the headlines, but determinism is the feature a model-risk team would ask about first. When a customer disputes a decision, or an auditor asks why a model said what it said, "run it again" should give the same answer. With most served language models, that is not guaranteed. Outputs can shift with the batch a request happened to share, so an output cannot be reproduced on demand. A production-grade engine that makes reproducibility a switch is a real step, and it is the kind of idea other open serving stacks can copy.
Open code answers how it runs. It does not answer whose model you are running.
The US order is the other half of the story. It applies to US federal civilian systems, not to banks in India or elsewhere. But procurement and risk teams everywhere will read it, and it separates two questions that often get blurred. One is whether the code is open and inspectable, which it is here. The other is whether the model's origin is acceptable to your regulator and your customers, which is a policy judgement no licence settles. And at 128 GPUs for an efficient deployment, this is infrastructure for a few large operators, not something most institutions will run themselves.
What I'd watch
- Whether batch invariance shows up in other open serving engines, which would make it usable with any model.
- Independent replication of DeepSeek's throughput and cost figures.
- Whether financial regulators start asking firms to record model origin alongside model version.
Sources
- deepseek-ai/DeepInfer (README) GitHub, Apache 2.0.
- DeepSeek open-sources DeepInfer, the inference framework behind its production services SiliconANGLE, 20 Oct 2026.
- U.S bans DeepSeek from government agency systems. CNBC, 20 Oct 2026.
- Notice of order regarding exclusion and removal of DeepSeek products from federal civilian executive branch IT systems Federal Register, 20 Oct 2026.