Deploying Ollama with Docker Compose: NVIDIA GPU Passthrough & Nginx Reverse Proxy

Running Large Language Models (LLMs) locally has become an indispensable capability for developers, security-conscious engineering teams, and homelab enthusiasts. Ollama (ollama/ollama) has established itself as the gold standard runtime for managing and serving open-source weights (such as Llama 3, Mistral, Gemma 2, and DeepSeek) via an intuitive, OpenAI-compatible REST API.

However, running Ollama in a bare Linux environment or relying on default desktop launchers often creates operational friction: models get deleted during container rebuilds, GPU acceleration fails silently due to missing container runtimes, and exposing the unauthenticated HTTP port 11434 directly to a local network poses serious security risks.

This production runbook details how to deploy Ollama with Docker Compose, pass through dedicated NVIDIA GPUs with the modern NVIDIA Container Toolkit (CDI), configure persistent high-speed volume storage, and secure the endpoint behind an Nginx reverse proxy with CORS headers.

1. Prerequisites: NVIDIA Container Toolkit Setup

Before configuring Docker Compose, your host system (Ubuntu 22.04/24.04 LTS or Debian 12) must have the host NVIDIA proprietary driver installed and the official NVIDIA Container Toolkit configured.

Step 1: Verify Host GPU Detection

nvidia-smi

Ensure your GPU model, driver version (535+ recommended), and CUDA version are displayed without errors.

Step 2: Install the NVIDIA Container Toolkit

# Configure the repository:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install packages:
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker daemon runtime:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 3: Test GPU Access Inside a Scratch Container

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If this command prints your GPU table, Docker is ready for accelerated LLM inference.

2. Production Docker Compose Configuration

Create an application directory (e.g., /opt/ollama-stack) and define the following docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    environment:
      - OLLAMA_KEEP_ALIVE=24h        # Keeps active model pinned in VRAM
      - OLLAMA_ORIGINS=*            # Enables cross-origin requests for Web UIs
      - OLLAMA_NUM_PARALLEL=2        # Concurrent request processing
    volumes:
      - /opt/ollama-data:/root/.ollama # Persistent storage for multi-GB model weights
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    networks:
      - internal_net

  nginx:
    image: nginx:alpine
    container_name: ollama-proxy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    depends_on:
      - ollama
    networks:
      - internal_net

networks:
  internal_net:
    driver: bridge

3. Hardened Nginx Reverse Proxy Configuration

Serving LLM responses requires specific proxy tuning: streaming responses (Server-Sent Events) can take hundreds of seconds for large generation tasks, which will cause standard Nginx proxy timeouts to fail prematurely.

Save the following as nginx.conf in the same directory:

events {
    worker_connections 1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile      on;

    upstream ollama_backend {
        server ollama:11434;
        keepalive 32;
    }

    server {
        listen 80;
        server_name ollama.internal.local;
        return 301 https://$host$request_uri;
    }

    server {
        listen 443 ssl;
        http2 on;
        server_name ollama.internal.local;

        ssl_certificate /etc/letsencrypt/live/ollama.internal.local/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/ollama.internal.local/privkey.pem;

        # Disable buffering to allow real-time token streaming
        proxy_buffering off;
        proxy_cache off;
        chunked_transfer_encoding on;

        # Extended timeouts for deep reasoning / long generations
        proxy_read_timeout 600s;
        proxy_connect_timeout 60s;
        proxy_send_timeout 600s;

        location / {
            proxy_pass http://ollama_backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # CORS headers for web-based frontends
            add_header 'Access-Control-Allow-Origin' '*' always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization' always;
        }
    }
}

4. Model Download & Verification Runbook

Launch the Stack

docker compose up -d

Verify GPU Utilization

# Monitor GPU processes
watch -n 1 nvidia-smi

Pull Your First Models

# Download Llama 3.1 (8B instruct)
docker exec -it ollama ollama pull llama3.1

# Download deep-reasoning or coding weights
docker exec -it ollama ollama pull qwen2.5-coder:7b

Test Inference via cURL

curl https://ollama.internal.local/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Explain the difference between TCP and UDP in one sentence.",
  "stream": false
}'

5. Common Pitfalls & Troubleshooting

Related Architecture & Developer Guides

• Caddy vs. Nginx: The Reverse Proxy Decision Tree for Self-Hosters in 2026 (Comparing SSL termination and reverse proxy architectures).
• Self-Hosting Uptime Kuma: The Complete Production Guide (Monitoring self-hosted container services).
• How to Fix 'Docker: permission denied while trying to connect to the Docker daemon socket' (Fixing Linux container permissions).