NVMe를 통해 LoRA 어댑터 스토리지를 관리하는 vLLM용 간단한 L7 프록시

A simple L7 proxy for vLLM that manages LoRA adapter storage via NVMe, routes requests, and pins workloads to nodes. loraplex sits between your clients and vLLM. It routes requests across a cluster using consistent hashing, manages LoRA adapter files on disk (fetching on demand from HuggingFace, S3, or HTTP, with LRU eviction), and provides node affinity through configurable hash keys. By default it hashes on the adapter name, but it can hash on any request header, enabling session pinning for prefix cache reuse, document-based routing for RAG workloads, or tenant isolation. vLLM's lora_filesystem_resolver reads adapter files from the same directory loraplex writes to. - Quick Start - How It Works - API - Examples and Deployment Modes - Architecture Details - Reference - Development - License go install github.com/shayonj/loraplex/cmd/loraplex@latest Or build from source: git clone https://github.com/shayonj/loraplex.git cd loraplex make build # Start vLLM with filesystem resolver pointed at loraplex's storage directory export VLLM_ALLOW_RUNTIME_LORA_UPDATING=true export VLLM_PLUGINS=lora_filesystem_resolver export VLLM_LORA_RESOLVER_CACHE_DIR=/mnt/nvme/loraplex python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --enable-lora --max-loras 200 --max-lora-rank 64 # Start loraplex (writes adapter files to the same directory vLLM reads from) ./bin/loraplex \ --listen :9090 \ --vllm-url http://localhost:8000 \ --dir /mnt/nvme/loraplex Point clients at localhost:9090 instead of localhost:8000 . Base model requests pass through unchanged. LoRA adapter requests trigger loraplex to fetch and store the adapter files, then proxy to vLLM which loads them from the shared directory. curl http://localhost:9090/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "your-org/your-lora-adapter", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50 }' On each node, pass --self and --peers : # Node 1 ./bin/loraplex --listen :9090 --self 10.0.0.1:9090 \ --vllm-url http://localhost:8000 \ --peers 10.0.0.1:9090,10.0.0.2:9090,10.0.0.3:9090 # Node 2 ./bin/loraplex --listen :9090 --self 10.0.0.2:9090 \ --vllm-url http://localhost:8000 \ --peers 10.0.0.1:9090,10.0.0.2:9090,10.0.0.3:9090 In K8s, loraplex discovers peers automatically via the Endpoints API so you don't need to list peers manually. See examples/k8s/ for full manifests including RBAC, headless Service, and Deployment. kubectl apply -f examples/k8s/manifests.yaml Key requirements: - A headless Service ( clusterIP: None ) selecting loraplex pods so the Endpoints API lists their IPs. - A ServiceAccount with RBAC permission to get Endpoints in the namespace. - Both the vLLM and loraplex containers must share the storage directory via an emptyDir volume. - vLLM environment variables ( VLLM_ALLOW_RUNTIME_LORA_UPDATING ,VLLM_PLUGINS ,VLLM_LORA_RESOLVER_CACHE_DIR ) must be set on the vLLM container. Use --config to load a YAML file instead of CLI flags. See config.example.yaml for all options. CLI flags override config file values. loraplex and vLLM share a directory on disk. loraplex writes adapter files there, and vLLM's filesystem resolver reads from there. loraplex does not modify requests to vLLM or inject paths. It ensures the files exist before proxying, and vLLM's resolver independently discovers them. loraplex writes ──▶ /mnt/nvme/loraplex/{adapter}/adapter_config.json /mnt/nvme/loraplex/{adapter}/adapter_model.safetensors vLLM reads ◀── VLLM_LORA_RESOLVER_CACHE_DIR=/mnt/nvme/loraplex vLLM manages adapters in GPU slots and CPU memory with its own LRU cache (--max-loras , --max-cpu-loras ). loraplex manages the layer below that: files on disk. It ensures adapter files are present in the shared directory before proxying requests to vLLM, handles on-demand fetching from remote origins, bounds disk usage with LRU eviction, and routes requests to the node that already has the adapter stored. Client request (model: "acme/summarizer-lora") │ ▼ loraplex (:9090) │ ├─ base model? ──▶ passthrough to vLLM (no storage needed) │ ├─ consistent hash ──▶ this node owns it? │ │ │ │ │ no │ yes │ ▼ ▼ │ forward to ensure adapter files exist in shared dir │ owner node │ │ │ on disk? ──▶ proxy to vLLM │ ▼ │ │ (owner runs │ not on disk │ same flow) ▼ │ fetch from HuggingFace/S3 │ write to shared dir │ │ │ ▼ │ proxy to vLLM │ ▼ vLLM (:8000) filesystem resolver finds adapter in shared dir loads adapter weights into CPU/GPU memory runs inference The diagram above shows the default behavior where hash_on is set to model . When hash_on uses a request header (e.g., header:X-Session-ID ), the same consistent hashing applies to all requests including base model requests. No adapter files are fetched for base model requests, but the routing still pins the request to a deterministic node. This is useful for workloads where landing on the same vLLM instance matters, like multi-turn conversations that benefit from vLLM's prefix cac