vLLM Production Deployment

Why Self-Host?

Data privacy: stays in your infrastructure

Cost: Llama 3 70B at $0.40/1M vs GPT-4o $2.50-10/1M

No rate limits

Break-even: roughly 5M tokens/month

vLLM: 20x More Throughput

vLLM implements PagedAttention, increasing GPU throughput 20-30x vs naive serving. Provides OpenAI-compatible API.

bash
pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --port 8000 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.85

Use with OpenAI SDK:

python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="na")
response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}]
)

GPU Requirements

Llama 3 8B: A10G 24GB minimum, A100 40GB recommended

Llama 3 70B: 2x A100 80GB minimum

Mistral 7B: A10G 24GB minimum

AWQ Quantization (75% Memory Reduction)

bash
python -m vllm.entrypoints.openai.api_server \
    --model casperhansen/llama-3-8b-instruct-awq \
    --quantization awq

AWQ 4-bit is near-lossless quality. Enables Llama 3 70B on single 80GB A100.

Kubernetes Config

Request 1 GPU per pod

readinessProbe initialDelaySeconds: 60

Mount HuggingFace cache as PersistentVolume

HPA on CPU utilization

Prometheus Monitoring

Key metrics:

request_latency_seconds

tokens_per_second

gpu_cache_usage_perc (keep below 90%)

num_requests_running