Groq API: Developer Guide and Quick Start 2026

Groq is a hardware and cloud services company focused on AI inference acceleration. Its custom LPU (Language Processing Unit) chip is specifically designed for large language model inference, offering significant advantages in low latency and high throughput. The Groq API provides an interface highly compatible with the OpenAI API, allowing developers to quickly integrate via standard HTTP requests or an SDK, leveraging the LPU's acceleration for tasks like text generation and conversation. This article will systematically cover using the Groq API, from environment setup and core usage to common integration patterns and a pitfall avoidance guide.

Environment Setup & Authentication

Obtaining an API Key

Visit console.groq.com to register an account and generate an API Key in the console. The key starts with gsk_; store it securely. Groq offers a free tier; specific limits and pricing are subject to the latest announcements on the official website. It is recommended to create a separate API Key for each project to facilitate permission management and usage tracking.

Installing the SDK

Groq provides an official Python SDK, installable via pip:

bash
pip install groq

The SDK reads the key from the environment variable GROQ_API_KEY by default. It is recommended to set this in a .env file or shell configuration:

bash
export GROQ_API_KEY="gsk_your_actual_key_here"

In your code, simply initialize the client:

python
from groq import Groq
client = Groq()  # Automatically reads GROQ_API_KEY

If you need to pass the key explicitly (e.g., for multi-account management), you can do:

python
client = Groq(api_key="gsk_your_actual_key_here")

Common Pitfall: Do not hardcode the API Key in your code, especially when committing to version control. Using environment variables or a key management service (e.g., HashiCorp Vault, AWS Secrets Manager) is a more secure choice. If a key is leaked, immediately revoke it in the console and generate a new one.

Core API Call: Text Generation

The core endpoint, chat.completions.create, supports both streaming and non-streaming output. Here is the simplest example:

python
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what an LPU is in English."}
    ],
    temperature=0.7,
    max_tokens=1024
)
print(response.choices[0].message.content)

Parameter Explanation

model: Specifies the model name. The list of available models is based on the official page at console.groq.com. Common models include:

- llama-3.3-70b-versatile (Llama 3.3 70B, general-purpose) - llama-3.1-8b-instant (Llama 3.1 8B, low latency) - openai/gpt-oss-120b (OpenAI open-source model 120B) - openai/gpt-oss-20b (OpenAI open-source model 20B)

messages: A list of conversation history, supporting system, user, and assistant roles.

temperature: Sampling temperature, range 0 to 2, default 1. Lower values make output more deterministic.

max_tokens: The maximum number of output tokens for this generation (excluding input). Setting a reasonable upper limit helps control costs and avoid abnormally long outputs.

Common Pitfall: Misspelling the model name is the most frequent issue for beginners. Groq's model name format is {provider}/{model}-{variant}, for example, llama-3.3-70b-versatile. Common errors include: omitting the version number (e.g., writing only llama-3.3), case errors (Groq model names are all lowercase), or using OpenAI model names (e.g., gpt-4) — Groq does not support OpenAI's closed-source models. Solution: Always copy the name from the model list at console.groq.com.

Streaming Output

The LPU's low-latency characteristic is particularly prominent in streaming output. By setting stream=True, the API returns results token by token, suitable for real-time display:

python
stream = client.chat.completions.create(
    model="llama-3.1-8b-instant",
    messages=[{"role": "user", "content": "Tell me a joke about programmers."}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

In a streaming response, each chunk's choices[0].delta contains the incremental content. Note: In streaming mode, response is an iterator and cannot be directly accessed for choices.

Common Pitfall: In streaming mode, chunk.choices[0].delta.content can be None (e.g., the last chunk only has a finish_reason). Always check for null values; otherwise, an AttributeError will be thrown. Additionally, when streaming, you need to concatenate all delta.content from all chunks yourself to get the complete response.

Using the OpenAI SDK for Compatibility

The Groq API is fully compatible with OpenAI's endpoint format, so you can directly use OpenAI's Python SDK, only needing to modify base_url and api_key:

bash
pip install openai

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ.get("GROQ_API_KEY")
)
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello, Groq!"}]
)
print(response.choices[0].message.content)

This method is suitable for projects already using the OpenAI SDK; you only need to replace the base_url to switch backends. Note: Groq's model list differs from OpenAI's, so you must use Groq-supported model names. This compatibility makes migration costs very low, especially suitable as a backend replacement in API integration tutorials.

Using cURL for Invocation

For non-Python environments, you can call the API directly via HTTP requests:

bash
curl https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{"role": "user", "content": "Hello, Groq!"}],
    "temperature": 0.7,
    "max_tokens": 256
  }'

For streaming output, add the "stream": true parameter and set the -N option to disable curl's buffering:

bash
curl -N https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instant",
    "messages": [{"role": "user", "content": "Tell me a joke."}],
    "stream": true
  }'

Common Pitfall: When using cURL, if -N is not set, streaming output will be buffered, causing the client to see the entire text instead of token-by-token output. Also, ensure the $GROQ_API_KEY environment variable is correctly set; otherwise, a 401 authentication error will be returned.

LPU Acceleration Principle & Advantages

Groq's LPU is a processor architecture specifically designed for sequential inference. Unlike the parallel matrix computation of GPUs, the LPU employs deterministic execution and a dataflow architecture, which significantly reduces token generation latency. In practice, its response speed is much faster than GPU-based inference services of comparable scale, making it especially suitable for scenarios requiring real-time interaction, such as chatbots, code completion, and real-time translation. Another advantage of the LPU is predictable latency — the latency variation for each request is minimal, which is crucial for production environments.

For scenarios requiring custom model deployment, refer to the model deployment guide to learn how to integrate Groq into existing inference pipelines.

Advanced Usage: Multi-turn Conversations & System Prompts

Groq supports multi-turn conversations by including the history in order within the messages list:

python
messages = [
    {"role": "system", "content": "You are a humorous assistant."},
    {"role": "user", "content": "What is recursion?"},
    {"role": "assistant", "content": "Recursion is when a function calls itself."},
    {"role": "user", "content": "Can you give me an example?"}
]
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=messages
)

Note: Long conversations consume more input tokens. It is recommended to periodically truncate the history or use token counting tools (e.g., tiktoken) to manage the context window.

Common Pitfall: In multi-turn conversations, if the messages list is too long (exceeding the model's context window), the API will return a 400 error. Solutions are: 1) Use the max_tokens parameter to limit output length; 2) Periodically truncate the history, keeping only the most recent turns; 3) Use a token counting tool to estimate input length.

Error Handling & Exponential Backoff Retry

In production environments, network jitter and rate limits are inevitable. For 429 (rate limit) and 5xx (server error) status codes, implement exponential backoff with jitter. Errors like 400 (parameter error) and 401 (authentication failure) are not worth retrying and should be raised directly:

python
import time
import random
from groq import Groq, RateLimitError, InternalServerError
client = Groq()
def chat_with_retry(messages, model="llama-3.3-70b-versatile", max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except (RateLimitError, InternalServerError) as e:
            if attempt == max_attempts - 1:
                raise
            wait = min(2 ** attempt, 30) + random.uniform(0, 1)
            print(f"Request failed ({type(e).__name__}), retrying in {wait:.1f}s...")
            time.sleep(wait)

Key points: The backoff time grows exponentially by powers of 2 with added random jitter to prevent a "retry storm" where many clients retry simultaneously; 2 ** attempt is capped at 30 seconds to prevent infinite waiting. For more comprehensive fault-tolerant designs (multi-provider degradation, circuit breakers), combine with related topics on the site.

Common Pitfalls & Avoidance Guide

1. Rate Limits & Quotas

The Groq free tier has request frequency limits (e.g., requests per minute, tokens per day). Exceeding these limits will return a 429 Too Many Requests error. Recommendations:

Implement exponential backoff retry in your code.

Monitor the X-RateLimit-* response headers.

Upgrade to a paid plan for production environments.

2. Key Management

Do not hardcode API Keys in your code; use environment variables or a key management service.

If a key is leaked, immediately revoke it in the console and generate a new one.

Separate keys for development and production environments.

3. Incorrect Handling of Streaming Output

In streaming mode, chunk.choices[0].delta.content can be None (e.g., the last chunk only has a finish_reason). Always check for null values.

4. Misspelling Model Names

Always copy the model name from the list at console.groq.com to avoid manual entry.

FAQ

Q: Does the Groq API support Function Calling? A: Some models support function calling; refer to the official documentation for specifics. Currently, models like llama-3.3-70b-versatile support it.

Q: What happens when the free tier quota is exhausted? A: Requests will be rejected due to quota exceeded (returning a 429-type rate limit error, with the response body specifying the quota type). You can check your usage in the console, wait for the quota to reset (per-minute/per-day quotas reset automatically), or upgrade to a paid plan.

Q: In which scenarios does Groq's LPU have a clear advantage over GPUs? A: The LPU excels in low latency and predictable latency, making it ideal for interactive scenarios like real-time conversation and code completion. For batch processing of large amounts of long text, GPUs may be more cost-effective.

Q: Can I deploy my own model on Groq? A: Currently, Groq only offers API services for pre-trained models and does not support custom model deployment. For private deployment, consider other platforms.

Q: How do I get the complete response when using streaming output? A: In streaming mode, you need to concatenate all delta.content from all chunks yourself. Alternatively, you can also make a non-streaming request to get the complete result (but this sacrifices real-time performance).

Q: Which model should I start with? A: For general chat and reasoning, llama-3.3-70b-versatile is the usual default. For latency-sensitive features (like autocomplete or real-time rewriting), llama-3.1-8b-instant trades some capability for speed. Either way, run a quick comparison on your own prompts before committing.

*Last updated: July 2026. Always verify against each tool's official docs.*