Context
Our client needed a large language model running on their own infrastructure rather than behind a commercial API.
Teams arrive at that requirement by different routes — data residency, per-token cost at volume, latency, or a regulatory constraint on sending data to a third party. The engineering consequence is the same whichever it is: you now own the serving problem, and the serving problem is where open-source LLM projects usually stall.
Problem
Getting an open-source model to answer one question on a workstation is a weekend. Getting it to answer many questions at once, reliably, is the actual engineering.
At 72 billion parameters the constraints bite immediately. The model has to fit in available memory before it serves anything. Every concurrent request competes for the same GPU. Batching raises throughput but costs individual latency. And a serving setup that behaves fine in testing can degrade sharply the moment real traffic arrives in bursts rather than evenly.
Our brief centred on concurrency: the system had to hold up when many requests arrived together, not just when they arrived politely one at a time.
Approach
The work split into three questions:
How the model is served. Configuration and serving architecture, so the model is loaded and addressable as a stable API rather than a script someone runs.
How concurrent requests are handled. Queuing and batching so that simultaneous requests are served without one starving the others.
How it behaves under sustained load. Performance stability, so throughput does not collapse and latency stays predictable as traffic rises.
What we built
Deployment and configuration of Qwen 2.5 72B, a serving architecture exposing it as an API, and concurrent request handling tuned for stability under load.
Outcome
The model runs on the client’s own infrastructure, exposed as a stable API and handling concurrent requests, with throughput and latency holding as load rises rather than degrading under burst traffic.
Throughput, latency and cost figures for this engagement are not published.
Related
Retrieval-Augmented Generation for an Education Platform · AI Engineering
