AI is evolving quickly, and software program engineers now not must memorize syntax. Nevertheless, considering like an architect and understanding the expertise that allows programs to run securely at scale is changing into more and more invaluable.
I additionally need to mirror on being in my function a 12 months now as an AI Options Engineer at Cisco. I work with clients every day throughout completely different verticals — healthcare, monetary providers, manufacturing, regulation corporations, and they’re all making an attempt to reply largely the identical set of questions:
- What’s our AI technique?
- What use circumstances really match our information?
- Cloud vs. on-prem vs. hybrid?
- How a lot will it value — not simply as we speak, however at scale?
- How can we safe it?
These are the actual sensible constraints that present up instantly when you attempt to operationalize AI past a POC.
Not too long ago, we added a Cisco UCS C845A to certainly one of our labs. It has 2x NVIDIA RTX PRO 6000 Blackwell GPUs, 3.1TB NVMe, ~127 allocatable CPU cores, and 754GB RAM. I made a decision to construct a shared inner platform on high of it — giving groups a constant, self-service atmosphere to run experiments, validate concepts, and construct hands-on GPU expertise.
I deployed the platform as a Single Node OpenShift (SNO) cluster and layered a multi-tenant GPUaaS expertise on high. Customers reserve capability via a calendar UI, and the system provisions an remoted ML atmosphere prebuilt with PyTorch/CUDA, JupyterLab, VS Code, and extra. Inside that atmosphere, customers can run on-demand inference, iterate on mannequin coaching and fine-tuning, and prototype manufacturing grade microservices.
This submit walks via the structure — how scheduling choices are made, how tenants are remoted, and the way the platform manages itself. The selections that went into this lab platform are the identical ones any group faces after they’re severe about AI in manufacturing.
That is the muse for enterprise AI at scale. Multi-agent architectures, self-service experimentation, safe multi-tenancy, cost-predictable GPU compute, all of it begins with getting the platform layer proper.
Preliminary Setup
Earlier than there’s a platform, there’s a naked steel server and a clean display.
Bootstrapping the Node
The node ships with no working system. Once you energy it on you’re dropped right into a UEFI shell. For OpenShift, set up sometimes begins within the Pink Hat Hybrid Cloud Console through the Assisted Installer. The Assisted Installer handles cluster configuration via a guided setup circulation, and as soon as full, generates a discovery ISO — a bootable RHEL CoreOS picture preconfigured on your atmosphere. Map the ISO to the server as digital media via the Cisco IMC, set boot order, and energy on. The node will cellphone residence to the console, and you’ll kick off the set up course of. The node writes RHCOS to NVMe and bootstraps. Inside a couple of hours you might have a operating cluster.
This workflow assumes web connectivity, pulling pictures from Pink Hat’s registries throughout set up. That’s not at all times an possibility. Lots of the clients I work with function in air-gapped environments the place nothing touches the general public web. The method there may be completely different: generate ignition configs domestically, obtain the OpenShift launch pictures and operator bundles forward of time, mirror all the things into an area Quay registry, and level the set up at that. Each paths get you to the identical place. The assisted set up is way simpler. The air-gapped path is what manufacturing seems to be like in regulated industries.
Configuring GPUs with the NVIDIA GPU Operator
As soon as the GPU Operator is put in (occurs robotically utilizing the assisted installer), I configured how the 2 RTX PRO 6000 Blackwell GPUs are introduced to workloads via two ConfigMaps within the nvidia-gpu-operator namespace.
The primary — custom-mig-config — defines bodily partitioning. On this case it’s a blended technique, that means GPU 0 is partitioned into 4 1g.24gb MIG slices (~24GB devoted reminiscence every), GPU 1 stays entire for workloads that want the complete ~96GB. MIG partitioning is actual {hardware} isolation. You get devoted reminiscence, compute models, and L2 cache per slice. Workloads will see MIG cases as separate bodily units.
The second — device-plugin-config — configures time-slicing, which permits a number of pods to share the identical GPU or MIG slice via fast context switching. I set 4 replicas per entire GPU and a pair of per MIG slice. That is what allows operating a number of inference containers facet by facet inside a single session.
Foundational Storage
The three.1TB NVMe is managed by the LVM Storage Operator (lvms-vg1 StorageClass). I created two PVCs as part of the preliminary provisioning course of — a quantity backing PostgreSQL and protracted storage for OpenShift’s inner picture registry.
With the OS put in, community stipulations met (DNS, IP allocation, all required A data) which isn’t coated on this article, GPUs partitioned, and storage provisioned, the cluster is prepared for the applying layer.
System Structure
This leads us into the primary subject: the system structure. The platform separates into three planes — scheduling, management, and runtime, with the PostgreSQL database as the one supply of fact.
Within the platform administration namespace, there are 4 at all times on deployments:
- Portal app: a single container operating the React UI and FastAPI backend
- Reconciler (controller): the management loop that repeatedly converges cluster state to match the database
- PostgreSQL: persistent state for customers, reservations, tokens, and audit historical past
- Cache daemon: a node-local service that pre-stages giant mannequin artifacts / inference engines so customers can begin rapidly (pulling a 20GB vLLM picture over company proxy can take hours)
A fast word on the event lifecycle, as a result of it’s straightforward to complicate transport Kubernetes programs. I write and check code domestically, however the pictures are constructed within the cluster utilizing OpenShift construct artifacts (BuildConfigs) and pushed to the inner registry. The deployments themselves simply level at these pictures.
The primary time a part is launched, I apply the manifests to create the Deployment/Service/RBAC. After that, most modifications are simply constructing a brand new picture in-cluster, then set off a restart so the Deployment pulls the up to date picture and rolls ahead:
oc rollout restart deployment/<deployment-name> -n <namespace>That’s the loop: commit → in-cluster construct → inner registry → restart/rollout.
The Scheduling Aircraft
That is the consumer going through entry level. Customers see the useful resource pool — GPUs, CPU, reminiscence, they decide a time window, select their GPU allocation mode (extra on this later), and submit a reservation.
GPUs are costly {hardware} with an actual value per hour whether or not they’re in use or not. The reservation system treats calendar time and bodily capability as a mixed constraint. The identical approach you’d e book a convention room, besides this room has 96GB of VRAM and prices significantly extra per hour.
Underneath the hood, the system queries overlapping reservations in opposition to pool capability utilizing advisory locks to stop double reserving. Primarily it’s simply including up reserved capability and subtracting it from whole capability. Every reservation tracks via a lifecycle: APPROVED → ACTIVE → COMPLETED, with CANCELED and FAILED as terminal states.
The FastAPI server itself is deliberately skinny. It validates enter, persists the reservation, and returns. It by no means talks to the Kubernetes API.
The Management Aircraft
On the coronary heart of the platform is the controller. It’s Python primarily based and runs in a steady loop on a 30-second cadence. You’ll be able to consider it like a cron job by way of timing, however architecturally it’s a Kubernetes-style controller chargeable for driving the system towards a desired state.
The database holds the specified state (reservations with time home windows and useful resource necessities). The reconciler reads that state, compares it in opposition to what really exists within the Kubernetes cluster, and converges the 2. There aren’t any concurrent API calls racing to mutate cluster state; only one deterministic loop making the minimal set of modifications wanted to achieve the specified state. If the reconciler crashes, it restarts and continues precisely the place it left off, as a result of the supply of fact (desired state) stays intact within the database.
Every reconciliation cycle evaluates 4 considerations so as:
- Cease expired or canceled periods and delete the namespace (which cascades cleanup of all assets inside it).
- Restore failed periods and take away orphaned assets left behind by partially accomplished provisioning.
- Begin eligible periods when their reservation window arrives — provision, configure, and hand the workspace to the consumer.
- Keep the database by expiring outdated tokens and imposing audit log retention.
Beginning a session is a multi-step provisioning sequence, and each step is idempotent, that means it’s designed to be safely re-run if interrupted halfway:
The reconciler is the solely part that talks to the Kubernetes API.
Rubbish assortment can be baked into the identical loop. At a slower cadence (~5 minutes), the reconciler sweeps for cross namespace orphans corresponding to stale RBAC bindings, leftover OpenShift safety context entries, namespaces caught in terminating, or namespaces that exist within the cluster however haven’t any matching database report.
The design assumption all through is that failure is regular. For instance, we had an influence provide failure on the node that took the cluster down mid-session and when it got here again, the reconciler resumed its loop, detected the state discrepancies, and self-healed with out guide intervention.
The Runtime Aircraft
When a reservation window begins, the consumer opens a browser and lands in a full VS Code workspace (code-server) pre-loaded with your entire AI/ML stack, and kubectl entry inside their session namespace.
Fashionable inference engines corresponding to vLLM, Ollama, TGI, and Triton are already cached on the node, so deploying a mannequin server is a one-liner that begins in seconds. There’s 600GB of persistent NVMe backed storage allotted to the session, together with a 20GB residence listing for notebooks and scripts, and a 300GB mannequin cache.
Every session is a totally remoted Kubernetes namespace, its personal blast radius boundary with devoted assets and 0 visibility into every other tenant’s atmosphere. The reconciler provisions namespace scoped RBAC granting full admin powers inside that boundary, enabling customers to create and delete pods, deployments, providers, routes, secrets and techniques — regardless of the workload requires. However there’s no cluster stage entry. Customers can learn their very own ResourceQuota to see their remaining price range, however they will’t modify it.
ResourceQuota enforces a tough ceiling on all the things. A runaway coaching job can’t OOM the node. A rogue container can’t fill the NVMe. LimitRange injects sane defaults into each container robotically, so customers can kubectl run with out specifying useful resource requests. There’s a proxy ConfigMap injected into the namespace so consumer deployed containers get company community egress with out guide configuration.
Customers deploy what they need — inference servers, databases, {custom} providers, and the platform handles the guardrails.
When the reservation window ends, the reconciler deletes the namespace and all the things inside it.
GPU Scheduling
Now the enjoyable half — GPU scheduling and truly operating hardware-accelerated workloads in a multi-tenant atmosphere.
MIG & Time-slicing
We coated the MIG configuration within the preliminary setup, however it’s value revisiting from a scheduling perspective. GPU 0 is partitioned into 4 1g.24gb MIG slices — every with ~24GB of devoted reminiscence, sufficient for many 7B–14B parameter fashions. GPU 1 stays entire for workloads that want the complete ~96GB VRAM for mannequin coaching, full-precision inference on 70B+ fashions, or something that merely doesn’t slot in a slice.
The reservation system tracks these as distinct useful resource varieties. Customers e book both nvidia.com/gpu (entire) or nvidia.com/mig-1g.24gb (as much as 4 slices). The ResourceQuota for every session laborious denies the alternative sort. In case you reserved a MIG slice, you bodily can’t request an entire GPU, even when one is sitting idle. In a blended MIG atmosphere, letting a session unintentionally devour the improper useful resource sort would break the capability math for each different reservation on the calendar.
Time-slicing permits a number of pods to share the identical bodily GPU or MIG slice via fast context switching. The NVIDIA gadget plugin advertises N “digital” GPUs per bodily gadget.
In our configuration, 1 entire GPU seems as 4 schedulable assets. Every MIG slice seems as 2.
What which means is a consumer reserves one bodily GPU and might run as much as 4 concurrent GPU-accelerated containers inside their session — a vLLM occasion serving gpt-oss, an Ollama occasion with Mistral, a TGI server operating a reranker, and a {custom} service orchestrating throughout all three.
Two Allocation Modes
At reservation time, customers select how their GPU price range is initially distributed between the workspace and consumer deployed containers.
Interactive ML — The workspace pod will get a GPU (or MIG slice) hooked up immediately. The consumer opens Jupyter, imports PyTorch, and has fast CUDA entry for coaching, fine-tuning, or debugging. Extra GPU pods can nonetheless be spawned through time-slicing, however the workspace is consuming one of many digital slots.
Inference Containers — The workspace is light-weight with no GPU hooked up. All time-sliced capability is obtainable for consumer deployed containers. With an entire GPU reservation, that’s 4 full slots for inference workloads.
There’s a actual throughput tradeoff with time-slicing, workloads share VRAM and compute bandwidth. For improvement, testing, and validating multi-service architectures, which is precisely what this platform is for, it’s the proper trade-off. For manufacturing latency delicate inference the place each millisecond of p99 issues, you’d use devoted slices 1:1 or entire GPUs.
GPU “Tokenomics”
One of many first questions within the introduction was: How a lot will it value — not simply as we speak, however at scale? To reply that, you need to begin with what the workload really seems to be like in manufacturing.
What Actual Deployments Look Like
Once I work with clients on their inference structure, no one is operating a single mannequin behind a single endpoint. The sample that retains rising is a fleet of fashions sized to the duty. You have got a 7B parameter mannequin dealing with easy classification and extraction, runs comfortably on a MIG slice. A 14B mannequin doing summarization and normal objective chat. A 70B mannequin for advanced reasoning and multi-step duties, and possibly a 400B mannequin for the toughest issues the place high quality is non-negotiable. Requests get routed to the suitable mannequin primarily based on complexity, latency necessities, or value constraints. You’re not paying 70B-class compute for a job a 7B can deal with.
In multi-agent programs, this will get extra attention-grabbing. Brokers subscribe to a message bus and sit idle till referred to as upon — a pub-sub sample the place context is shared to the agent at invocation time and the pod is already heat. There’s no chilly begin penalty as a result of the mannequin is loaded and the container is operating. An orchestrator agent evaluates the inbound request, routes it to a specialist agent (retrieval, code technology, summarization, validation), collects the outcomes, and synthesizes a response. 4 or 5 fashions collaborating on a single consumer request, every operating in its personal container throughout the identical namespace, speaking over the inner Kubernetes community.
Community insurance policies add one other dimension. Not each agent ought to have entry to each instrument. Your retrieval agent can discuss to the vector database. Your code execution agent can attain a sandboxed runtime. However the summarization agent has no enterprise touching both, it receives context from the orchestrator and returns textual content. Community insurance policies implement these boundaries on the cluster stage, so instrument entry is managed by infrastructure, not utility logic.
That is the workload profile the platform was designed for. MIG slicing permits you to proper measurement GPU allocation per mannequin, a 7B doesn’t want 96GB of VRAM. Time-slicing lets a number of brokers share the identical bodily gadget. Namespace isolation retains tenants separated whereas brokers inside a session talk freely. The structure immediately helps these patterns.
Quantifying It
To maneuver from structure to enterprise case, I developed a tokenomics framework that reduces infrastructure value to a single comparable unit: value per million tokens. Every token carries its amortized share of {hardware} capital (together with workload combine and redundancy), upkeep, energy, and cooling. The numerator is your whole annual value. The denominator is what number of tokens you really course of, which is totally a operate of utilization.
Utilization is probably the most highly effective lever on per-token value. It doesn’t cut back what you spend, the {hardware} and energy payments are mounted. What it does is unfold these mounted prices throughout extra processed tokens. A platform operating at 80% utilization produces tokens at practically half the unit value of 1 at 40%. Identical infrastructure, dramatically completely different economics. Because of this the reservation system, MIG partitioning, and time-slicing matter past UX — they exist to maintain costly GPUs processing tokens throughout as many out there hours as potential.
As a result of the framework is algebraic, you may also clear up within the different course. Given a recognized token demand and a price range, clear up for the infrastructure required and instantly see whether or not you’re over-provisioned (burning cash on idle GPUs), under-provisioned (queuing requests and degrading latency), or right-sized.
For the cloud comparability, suppliers have already baked their utilization, redundancy, and overhead into per-token API pricing. The query turns into: at what utilization does your on-prem unit value drop beneath that charge? For constant enterprise GPU demand, the sort of steady-state inference site visitors these multi-agent architectures generate, on-prem wins.
Cloud token prices in multi-agent environments scale parabolically.
Nevertheless, for testing, demos, and POCs, cloud is cheaper.
Engineering groups usually must justify spend to finance with clear, defensible numbers. The tokenomics framework bridges that hole.
Conclusion
At the start of this submit I listed the questions I hear from clients continually — AI technique, use-cases, cloud vs. on-prem, value, safety. All of them finally require the identical factor: a platform layer that may schedule GPU assets, isolate tenants, and provides groups a self-service path from experiment to manufacturing with out ready on infrastructure.
That’s what this submit walked via. Not a product and never a managed service, however an structure constructed on Kubernetes, PostgreSQL, Python, and the NVIDIA GPU Operator — operating on a single Cisco UCS C845A with two NVIDIA RTX PRO 6000 Blackwell GPUs in our lab. It’s a sensible start line that addresses scheduling, multi-tenancy, value modeling, and the day-2 operational realities of conserving GPU infrastructure dependable.
This isn’t as intimidating because it seems to be. The tooling is mature, and you’ll assemble a cloud-like workflow with acquainted constructing blocks: reserve GPU capability from a browser, drop into a totally loaded ML workspace, and spin up inference providers in seconds. The distinction is the place it runs — on infrastructure you personal, beneath your operational management, with information that by no means leaves your 4 partitions. In follow, the barrier to entry is usually decrease than leaders count on.
Scale this to a number of Cisco AI Pods and the scheduling aircraft, reconciler sample, and isolation mannequin carry over immediately. The inspiration is similar.
In case you’re working via these identical choices — methods to schedule GPUs, methods to isolate tenants, methods to construct the enterprise case for on-prem AI infrastructure, I’d welcome the dialog.
I’m an AI Options Engineer at Cisco, specializing in enterprise AI infrastructure. Attain out at [email protected].
