Serverless AI: Lambda, Cloudflare Workers & Edge Inference

When to Use Serverless for AI

Serverless AI excels when: traffic is unpredictable (spikes and quiet periods), you want zero infrastructure management, cold start latency is acceptable, inference runs in under 15 minutes (Lambda limit), and model size fits in memory limits (10GB for Lambda).

Not ideal for: GPU inference requiring dedicated GPUs, large models (>10GB), very high sustained throughput (server is more cost-effective), streaming inference requiring persistent connections.

AWS Lambda for ML Inference

Container-Based Lambda for ML

Lambda container images support up to 10GB—sufficient for many ML models. Use Python 3.11 base image with slim PyTorch or ONNX Runtime. Package model weights inside the container or load from S3 at cold start.

Dockerfile: FROM public.ecr.aws/lambda/python:3.11. Install only required packages (torch-cpu for CPU inference). Copy model weights. Set handler function. Build and push to ECR.

Lambda configuration: memory 3008MB (affects CPU allocation), timeout 30 seconds for inference, reserve concurrency to limit parallel invocations (cost control), use Provisioned Concurrency to eliminate cold starts for critical workloads.

Cold start optimization: minimize package size (torch CPU is 600MB vs 2GB+ for GPU), use lazy loading for model weights, implement warmup pings every 5 minutes via EventBridge.

Inference Patterns on Lambda

Synchronous: API Gateway → Lambda → return prediction. Best for real-time use cases with <15s inference time. Configure API Gateway timeout to match Lambda timeout.

Asynchronous: client → SQS → Lambda → store results in DynamoDB → client polls. Best for long-running inference, batch processing, workflows that don't need immediate response.

Cost Optimization

Lambda pricing: $0.0000166667 per GB-second. 3GB Lambda, 1-second inference = $0.00005 per request = $50 per million requests. Compare to dedicated EC2: m5.large ($0.096/hour) handles ~100 requests/second = $0.000027 per request. Break-even: if you have sustained load over 500,000 requests/day, dedicated compute may be cheaper.

Cloudflare Workers AI

Cloudflare Workers AI runs inference at the edge across 300+ PoPs globally. Models available: text generation (Llama 3), text embeddings, image classification, speech recognition, translation. No GPU to manage—Cloudflare provides the inference infrastructure.

Workers AI example: fetch the AI binding, call ai.run with the model ID and messages array, return the JSON response. Latency: typically 100-300ms globally due to edge distribution.

Limitations: model selection limited to Cloudflare's catalog, no custom model deployment (yet), throughput limits per account.

Vercel AI SDK Edge Functions

Vercel's AI SDK enables streaming LLM responses from edge functions: import streamText from 'ai', import the OpenAI provider, call streamText with model and messages, return the resulting text stream response. Edge functions deploy to Vercel's 100+ edge locations globally.

Supports: OpenAI, Anthropic, Google, Mistral, and custom providers. Built-in streaming, token counting, and error handling.

ONNX Runtime for Portable Inference

Export models to ONNX format for framework-independent inference. ONNX Runtime supports: CPUs (x86, ARM), GPUs (CUDA, DirectML, CoreML), WebAssembly (browser inference), and edge devices.

Export PyTorch model to ONNX: torch.onnx.export with dummy input tensor. Optimize with onnxruntime-tools: reduce model precision, fuse operators. Deploy with ONNXRuntime.InferenceSession—no PyTorch dependency needed.

ONNX on Lambda: 10x smaller package than PyTorch, 2-3x faster CPU inference. Ideal for classic ML models (sklearn, XGBoost) and smaller neural networks.

Edge AI: WebAssembly and TensorFlow.js

Browser Inference

TensorFlow.js runs models directly in the browser: no server round-trip needed, works offline, user data never leaves device, zero infrastructure cost. Use cases: real-time pose detection, object detection from webcam, on-device text classification.

Load model with tf.loadLayersModel or tf.loadGraphModel. Run inference with model.predict. Use the WebGL backend for GPU acceleration in supported browsers.

WebAssembly for Server-Edge

WASI (WebAssembly System Interface) enables running ONNX models in WebAssembly at near-native speed. Wasmtime or wasmer provides the runtime. Deploy to edge platforms that support WASM (Cloudflare Workers, Fastly Compute@Edge, Deno Deploy).

Choosing Your AI Deployment Strategy

FactorServer (EC2/GKE)Serverless (Lambda)Edge (CF Workers)

LatencyLow (if same region)Medium (cold start)Very low (global edge) Cost at scaleBestGoodPremium Cost at low trafficWastefulBestGood Model sizeAny<10GB<1GB Custom modelsYesYesNo (catalog only) GPU inferenceYesLimitedNo

Edge AI shines for: real-time personalization, content moderation at scale, globally distributed applications, and privacy-sensitive inference where data shouldn't leave the user's region.