In conversations with our users, we kept hearing the same questions:
With only a few GPU servers on hand, building a complex infrastructure stack just isn't worth it.
AI and GPU operations are complicated — if a platform isn't simple and intelligent enough, administrators simply won't use it.
Many platforms claim heterogeneous support, yet never clarify which card can run which model — so "heterogeneous" never actually lands.
And when you don't have many cards, GPU partitioning only solves small-model deployment; how do you improve utilization for large models?
GPUWorks was built to answer exactly these questions.
It's just 40+ MB, installs with a single command, and doesn't depend on Kubernetes. Its built-in AI assistant can directly participate in fault diagnosis, and every scheduling decision comes with its rationale.
We enumerate every "model × chip" combination — nearly 400 models and 1,000+ ready-to-use launch configurations out of the box — and use sleep-and-wake to let multiple large models time-share the same set of GPUs.
Small in size doesn't mean simple in functionality, and it certainly doesn't mean weak in capability.
In real-machine testing:
A single GPUWorks Server managed 1,003 GPU servers and brought up 500+ inference service replicas in 4 minutes; it output 69,000 tokens per second while the Server consumed only 9 CPU cores and 1.6 GB of memory; waking a hibernating DeepSeek-V4.1-Flash took as little as 3.6 seconds.
From GPU hardware health to per-request latency, everything is covered by complete monitoring and analytics; from host RDMA networking to Ascend Lingqu HCCS, everything is automatically discovered and managed; from placement decisions to model switching, every scheduling step is fully traceable.
GPUWorks is a lightweight, enterprise-grade AI inference platform that brings compute from different vendors and sources under unified management. Through model-hardware matching, VRAM evaluation, and template-based deployment, it delivers model services to business applications via an OpenAI-compatible API.
GitHub repository: GitHub - flitai-io/gpuworks
Enterprises and developers can obtain the code, use and modify the platform, and contribute inference templates and chip adaptations.
01 / Three Core Strengths
Three core strengths that put compute to work for your business, faster
Highlight 1: Ultra-Lightweight
40+ MB to get started
60 seconds to complete platform installation and deployment
Building an enterprise AI inference platform starts with a 40+ MB installation package. GPUWorks requires no pre-built Kubernetes — one command, 60 seconds, and the platform is installed and deployed. Administrators can open the management console in a browser, connect compute nodes, and view compute resources, getting everything ready for model deployment. For enterprise intranets, installation packages, model files, and runtime images can all be imported offline, so you can build at your own pace, in your own environment.
The platform installs fast — and there are ready-made solutions for getting models online too. GPUWorks provides 1,000+ launch configurations for nearly 400 models, packaging images, inference engines, weight versions, and startup parameters into reusable, one-click deployment plans. Administrators select templates based on the model and available hardware, and allocate resources with VRAM evaluation — eliminating the repetitive work of preparing environments and tweaking configurations.
Highlight 2: Time-Sharing
One group of GPUs, multiple large models
Policy-driven, highly efficient automatic switching
The same set of GPUs, efficiently time-shared across multiple large models. GPUWorks lets multiple large models take turns using the same group of GPUs on demand — no need to permanently dedicate a set of compute to every model. Enterprises can deploy more models on existing hardware, offer model choices to different business lines, and cut the resource investment that multi-model deployment would otherwise require.
Configure reuse policies, and the platform coordinates model switching automatically. Administrators can set priority, idle-yield, or scheduled-rotation policies based on business needs; the platform then orchestrates model sleep, resource release, and wake-up accordingly. For models that require multiple GPUs working in concert, resources are yielded and reclaimed in whole GPU groups — freeing administrators from repeatedly stopping and starting models and adjusting resources by hand.
From over ten minutes down to about 4 seconds — a roughly 200x improvement in model service wake-up efficiency. With L1 sleep, which keeps model weights in host memory, GPUWorks doesn't need to restart the engine or reload weights from storage on wake-up, dramatically reducing the waiting cost of alternating between models. In tests on 8 × H20, three large models — each deployed with 8-way parallelism — took turns using the same GPU group: cold start reading weights from NFS took 16–17.5 minutes, while in-memory wake-up restored service in about 4 seconds.
Highlight 3: Unified Management at 1,000-Node Scale
1,000+ nodes of compute
A platform that grows with your business
From 0 to 1,000+ nodes — unified compute management, scaled on demand. Enterprises can start building with existing equipment and keep connecting compute nodes as the business grows — viewing resources, deploying models, and tracking operational status all on the same platform. From trying out a handful of models to sustained calls from multiple business lines, administrators can orchestrate compute and model services in one place.
Compute comes from many places; deployment and management stay centralized on one platform. GPUWorks supports NVIDIA and AMD chips, and is broadly adapted to domestic compute — covering Huawei Ascend, T-Head PPU, Hygon DCU, and many other domestic AI chips — giving enterprises more options for adopting domestic compute. Bare metal, VMs, and cloud containers across on-premises data centers, private clouds, and public clouds can all be brought under unified management. The platform filters available hardware against model compatibility requirements, helping enterprises consolidate existing equipment and top up with cloud resources for the card types and capacity they need — without maintaining separate deployment tooling for each environment.
Break through single-machine capacity limits and scale service capability with business load. Multi-machine parallelism lets multiple machines jointly host large models that no single machine could run, expanding the range of deployable models. Multi-replica distribution spreads requests across multiple instances, boosting concurrency and easing queuing pressure during peak load. For suitable models, PD disaggregation deploys input processing and content generation separately, making it easy to provision and scale resources for each stage according to its actual load — reducing resource mismatch and targeting compute investment at the real bottlenecks. Administrators can also combine service status, request metrics, and logs to judge whether scaling actually improved responsiveness, and continuously fine-tune resource allocation.
02 / Built for Business
Six core application scenarios that turn platform capabilities into business value
Enterprise private inference platform: one set of model services shared across applications
In an enterprise's own environment, uniformly support knowledge retrieval, intelligent Q&A, voice interaction, and content generation. GPUWorks supports offline import of models and images, inter-node synchronization, and template-based deployment — making it easy to build an inference platform on an intranet or in environments with restricted external access. Different types of models can be continuously onboarded, providing a unified access point for all business applications and eliminating duplicate building and maintenance across departments.
Cross-cloud and multi-vendor compute coordination: organize scattered devices into usable resources
Consolidate existing compute and bring in cloud resources on demand to meet diverse hardware and capacity requirements. Facing resources scattered across on-premises data centers, private clouds, and rented cloud GPUs, GPUWorks uniformly connects bare metal, VMs, and cloud containers, selecting hardware according to model compatibility conditions. Administrators can arrange model deployments across different environments on the same platform, making the most of existing investments while topping up compute as the business requires.
New model validation under limited compute: balancing iteration with business continuity
New models keep emerging — and enterprises can keep validating them on existing compute. Administrators plan deployments using inference templates and VRAM evaluation, freeing up resources with dGPU card-sharing and vLLM sleep. For multi-replica services, only some replicas can be put to sleep while others keep serving — so trying out new models and running the business can share the same compute, reducing the pressure to buy new hardware for every validation cycle.
Large-model and high-concurrency service scaling: models fit, requests are handled
Choose the right scaling approach based on model size and business call volume. GPUWorks supports multi-machine parallelism, multi-replica, and PD disaggregation, and selects nodes based on idle cards, chip architecture, and shared networking — with RDMA or Ascend HCCS communication configured. Before scaling, administrators can use the built-in stress testing to observe throughput and latency, judge how many requests the current configuration can carry, and then decide how much resource to add.
Inference usage accounting and compute efficiency analysis: see where resources are going
Which applications call the most? Which resources sit idle? Let the data drive investment decisions. GPUWorks tracks token usage and cache hit rates by API Key, model, and deployment, and combines this with GPU occupancy time, card-hours per million output tokens generated, and idle-GPU analysis — helping enterprises run internal usage accounting, identify reclaimable resources, and build the case for future expansion.
Online inference service maintenance and fault recovery: keep maintenance and troubleshooting orderly
From node maintenance to service anomalies, reduce the manual work of machine-by-machine inspection and recovery. When services need to be migrated off a node, the platform adds replacement replicas first, then lets the original replicas finish their in-flight requests before exiting. If a node is unreachable for an extended period, the platform automatically selects a qualifying node to restore the service. With request tracing, logs, and AI diagnosis, administrators can pinpoint slow requests and instance failures — understanding both the cause and the path to resolution.
03 / Make Any GPU Work
Turn every investment in compute into business value, faster
From a 40+ MB lightweight start, to second-level large-model switching, to unified management of 1,000+ nodes — GPUWorks aims to make it easier for enterprises to get started, and to keep expanding their AI applications with confidence: put existing compute to work, put idle time to use, and deliver model services to the businesses that truly need them.
Open source opens this path to everyone. Enterprises can inspect and extend the platform; developers can contribute templates; hardware partners can refine adaptations — together, getting more models running on more compute.
Join the GPUWorks open-source community
Try GPUWorks, share feedback, or contribute inference templates and chip adaptations — your practice can help more teams make the most of their models and compute.
GitHub repository: GitHub - flitai-io/gpuworks: A ~45 MB MaaS platform: one cluster across public and private clouds, on any mix of accelerators.
Scan the QR code below to join the GPUWorks community — exchange deployment experience with developers, enterprise technical teams, and hardware partners, discuss product requirements, and help improve GPUWorks together.
Livestream Reservation
1,000 GPU servers, live and serving in 1 hour!
Deployment must be fast, management must be simple — and even at a thousand-node scale, it should be effortless. Friday, November 6, 14:00 — tune in to the "ZStack Cloud Computing" livestream. From compute in place to models running, join us to uncover how large-scale AI services go live, fast!