Deploy Llama 3.2 3B on NVIDIA Jetson Orin

Overview

Run Llama 3.2 3B directly on NVIDIA Jetson Orin for robotics and edge AI. Local inference offers privacy, zero latency, and no ongoing API costs.

Specs: Ampere GPU · 8GB

Installation

bash
Install Ollama — easiest local inference runtime
curl -fsSL https://ollama.com/install.sh | sh
Verify installation
ollama --version

Download Model

bash
Pull Llama 3.2 3B (downloads GGUF quantized weights automatically)
ollama pull llama-32-3b
Run interactive chat
ollama run llama-32-3b
Start API server
ollama serve
API available at http://localhost:11434

Python Integration

python
import httpx
from typing import Iterator
class LocalAI:
    """Interface to local Llama 3.2 3B running on NVIDIA Jetson Orin."""
    
    BASE_URL = "http://localhost:11434"
    MODEL = "llama-32-3b"
    
    def chat(self, message: str, system: str = "") -> str:
        """Single-turn chat."""
        resp = httpx.post(
            f"{self.BASE_URL}/api/chat",
            json={
                "model": self.MODEL,
                "messages": [
                    {"role": "system", "content": system},
                    {"role": "user", "content": message}
                ],
                "stream": False
            },
            timeout=120
        )
        resp.raise_for_status()
        return resp.json()["message"]["content"]
    
    def stream(self, message: str) -> Iterator[str]:
        """Streaming chat for real-time output."""
        with httpx.stream(
            "POST",
            f"{self.BASE_URL}/api/chat",
            json={"model": self.MODEL, "messages": [{"role": "user", "content": message}], "stream": True},
            timeout=120
        ) as r:
            for line in r.iter_lines():
                if line:
                    import json
                    chunk = json.loads(line)
                    if not chunk.get("done"):
                        yield chunk["message"]["content"]
Usage
ai = LocalAI()
response = ai.chat("Help me with robotics and edge AI")
print(response)
Streaming
for token in ai.stream("Explain robotics and edge AI step by step"):
    print(token, end="", flush=True)

Custom Modelfile

bash
Create optimized configuration for robotics and edge AI
cat > Modelfile << 'MODELEOF'
FROM llama-32-3b
PARAMETER num_ctx 4096
PARAMETER temperature 0.7
PARAMETER top_p 0.9
SYSTEM "You are an AI assistant specialized in robotics and edge AI. You run locally on NVIDIA Jetson Orin. Be concise, accurate, and helpful."
MODELEOF
ollama create robotics-and-edge-AI-assistant -f Modelfile
ollama run robotics-and-edge-AI-assistant

Performance Profile

MetricValue

HardwareAmpere GPU Memory8GB Speed10-40 tokens/sec (CPU) / 40-100+ tok/s (GPU) First token<200ms (GPU) / <1s (CPU) Context4096-32768 tokens Cost$0 (after hardware)

Production Setup with FastAPI

python
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI(title="NVIDIA Jetson Orin AI API")
ai = LocalAI()
class ChatRequest(BaseModel):
    message: str
    system: str = ""
class ChatResponse(BaseModel):
    response: str
    model: str
    device: str
@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(req: ChatRequest):
    response = ai.chat(req.message, req.system)
    return ChatResponse(response=response, model="Llama 3.2 3B", device="NVIDIA Jetson Orin")
@app.get("/health")
async def health():
    return {"status": "ok", "model": "Llama 3.2 3B", "device": "NVIDIA Jetson Orin"}

Troubleshooting

Slow inference: Switch to Q4_K_M quantization, reduce context window Out of memory: Use smaller model or Q3_K_S quant GPU not used: Install CUDA/Metal drivers, check ollama logs High latency: Warm up model by sending a dummy request on startup

Resources

Ollama library: https://ollama.com/library

GGUF format: https://github.com/ggerganov/llama.cpp

Hardware guide: https://ollama.com/blog/hardware-recommendations