Run copilot isn't a magic thing, i was asked how to run this on a mac as example, alternatively you can place a copilot server within the office for everyone usage.

That's the same way how OpenAI, Google … and all other work, what is different? they have better hardware, more developer, more budget. The only difference is if you ask your pilot about special things, like how you will start with as example money laundering, you get a valid answer

The solution is completely Open Source you’re free to modify it yourself, it combines local running AI Stack and the Home Assistant. Who do not know Home Assistant see https://www.home-assistant.io/ It runs completely from Docker container, if you want remove it, just delete the container. It will not pollute you system.

Note: This will require some basic understanding on several technologies, like docker networking and Linux, as a Systemadmin Devops or Linux Engineer you should understand what i’m talking about.

Feel free to contribute!

Requirements

Installed Gnoppix 24.8 or Debian 12 and some manual steps.

Assumed you have a modern computer as example:

GPU: recom NVIDIA 3090i or better with 12+ GB RAM

CPU 4-24 cores ( if you have not such a powerful GPU, you can run it also by CPU) as faster as better

SSD : Samsung EVO 980 2GB

RAM: 64+GB

Local setuped domain : gnoppix.local (feel free to change it whatever you prefer)

Software Overview:

Here’s the tech stack I’m using:

I've warned you repeatedly that you'll need a substantial amount of storage space. This guide isn't sponsored by NVIDIA; I've just been using their hardware as it was available. I've tried contacting AMD but haven't received any responses, so I recommend using NVIDIA for now.

For this general setup, I've used Docker's internal naming conventions. If needed, you can separate different services, for example, running Ollama in a Kubernetes cluster for handling more concurrent requests. This structure might change in the future. I initially developed this solution as a freelance project for a friend's company

You need to have docker installed and running

Structure

Running everything from the /opt/copilot folder it looks like as followed:

Stacks live in /opt/gnoppix-co-pilot

Here is the folder structure. Most sub folders are created when binding volumes.

├── gnoppix-co-pilot
│   ├── .env
│   ├── compose.yaml
│   ├── ollama
│   ├── open-webui
│   ├── searxng
│   ├── stable-diffusion-webui-docker
│   └── whisper
├── gnoppix-home-assistant
│   ├── compose.yaml
│   ├── faster-whisper
│   ├── home-assistant
│   └── wyoming-piper

Step1.)

Create a username and password for ollama

echo $(htpasswd -nB ollamauser) | sed -e s/\\$/\\$\\$/g

You’ll then want to place this in your .env using the OLLAMA_API_CREDENTIALS variable. This is then used in the ollama service in your compose file.

Basic Auth Hash

If you want to create a hash value for Basic Auth . You’ll need to use the credential from above.

echo 'ollamauser:ollamapass!' | base64

2.) Install NVIDIA CTK

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository:

sudo apt-get update

Install the NVIDIA Container Toolkit packages

sudo apt-get install -y nvidia-container-toolkit

Configure the container runtime by using the nvidia-ctk command and restart docker

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

3.) Build the container

cd /opt/gnoppix-co-pilot

# docker compose up -d --build

or just run

cd /opt/gnoppix-co-pilot

docker compose up -d --build --force-recreate --remove-orphans

4.) Create docker network for traefik (skip the step if you already using traefik) change the network if you already use 172.16.10.x

docker network create traefik

docker network create -d gnoppixvlan \
--subnet=172.16.10.0/24 \
--gateway=172.16.10.1 \
-o parent=eth1 \
iot_gnoppixvlan

compose-traefik.yml looks like this:

services:

# Ollama

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - OLLAMA_KEEP_ALIVE=24h
      - ENABLE_IMAGE_GENERATION=True
      - COMFYUI_BASE_URL=http://stable-diffusion-webui:7860
    networks:
      - traefik
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./ollama:/root/.ollama
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.ollama.rule=Host(`ollama.gnoppix.local`)"
      - "traefik.http.routers.ollama.entrypoints=https"
      - "traefik.http.routers.ollama.tls=true"
      - "traefik.http.routers.ollama.tls.certresolver=cloudflare"
      - "traefik.http.routers.ollama.middlewares=default-headers@file"
      - "traefik.http.routers.ollama.middlewares=ollama-auth"
      - "traefik.http.services.ollama.loadbalancer.server.port=11434"
      - "traefik.http.routers.ollama.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=${OLLAMA_API_CREDENTIALS}"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

# open web ui
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    networks:
      - traefik
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - 'OLLAMA_BASE_URL=http://ollama:11434'
      - ENABLE_RAG_WEB_SEARCH=True
      - RAG_WEB_SEARCH_ENGINE=searxng
      - RAG_WEB_SEARCH_RESULT_COUNT=3
      - RAG_WEB_SEARCH_CONCURRENT_REQUESTS=10
      - SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./open-webui:/app/backend/data
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.open-webui.rule=Host(`chat.gnoppix.local`)"
      - "traefik.http.routers.open-webui.entrypoints=https"
      - "traefik.http.routers.open-webui.tls=true"
      - "traefik.http.routers.open-webui.tls.certresolver=cloudflare"
      - "traefik.http.routers.open-webui.middlewares=default-headers@file"
      - "traefik.http.services.open-webui.loadbalancer.server.port=8080"
    depends_on:
      - ollama
    extra_hosts:
      - host.docker.internal:host-gateway

  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    networks:
      - traefik
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./searxng:/etc/searxng
    depends_on:
      - ollama
      - open-webui
    restart: unless-stopped

# stable diffusion

  stable-diffusion-download:
    build: ./stable-diffusion-webui-docker/services/download/
    image: comfy-download
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./stable-diffusion-webui-docker/data:/data

  stable-diffusion-webui:
    build: ./stable-diffusion-webui-docker/services/comfy/
    image: comfy-ui
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - CLI_ARGS=
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./stable-diffusion-webui-docker/data:/data
      - ./stable-diffusion-webui-docker/output:/output

    stop_signal: SIGKILL
    tty: true
    deploy:
      resources:
        reservations:
          devices:
              - driver: nvidia
                device_ids: ['0']
                capabilities: [compute, utility]
    restart: unless-stopped
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.stable-diffusion.rule=Host(`stable-diffusion.gnoppix.local`)"
      - "traefik.http.routers.stable-diffusion.entrypoints=https"
      - "traefik.http.routers.stable-diffusion.tls=true"
      - "traefik.http.routers.stable-diffusion.tls.certresolver=cloudflare"
      - "traefik.http.services.stable-diffusion.loadbalancer.server.port=7860"
      - "traefik.http.routers.stable-diffusion.middlewares=default-headers@file"

# whisper
  mongo:
    image: mongo
    env_file:
      - .env
    networks:
      - traefik
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./whisper/db_data:/data/db
      - ./whisper/db_data/logs/:/var/log/mongodb/
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - MONGO_INITDB_ROOT_USERNAME=${DB_USER:-whisper}
      - MONGO_INITDB_ROOT_PASSWORD=${DB_PASS:-whisper}
    command: ['--logpath', '/var/log/mongodb/mongod.log']

  translate:
    container_name: whisper-libretranslate
    image: libretranslate/libretranslate:latest-cuda
    env_file:
      - .env
    networks:
      - traefik
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./whisper/libretranslate/data:/home/libretranslate/.local/share
      - ./whisper/libretranslate/cache:/home/libretranslate/.local/cache
    user: root
    tty: true
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - LT_DISABLE_WEB_UI=True
      - LT_LOAD_ONLY=${LT_LOAD_ONLY:-en,de,fr,es}
      - LT_UPDATE_MODELS=True
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]

  whisper:
    container_name: whisper
    pull_policy: always
    image: pluja/whishper:latest-gpu
    env_file:
      - .env
    networks:
      - traefik
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./whisper/uploads:/app/uploads
      - ./whisper/logs:/var/log/whishper
      - ./whisper/models:/app/models
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whisper.rule=Host(`whisper.gnoppix.local`)"
      - "traefik.http.routers.whisper.entrypoints=https"
      - "traefik.http.routers.whisper.tls=true"
      - "traefik.http.routers.whisper.tls.certresolver=cloudflare"
      - "traefik.http.services.whisper.loadbalancer.server.port=80"
      - "traefik.http.routers.whisper.middlewares=default-headers@file"
    depends_on:
      - mongo
      - translate
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - PUBLIC_INTERNAL_API_HOST=${WHISHPER_HOST}
      - PUBLIC_TRANSLATION_API_HOST=${WHISHPER_HOST}
      - PUBLIC_API_HOST=${WHISHPER_HOST:-}
      - PUBLIC_WHISHPER_PROFILE=gpu
      - WHISPER_MODELS_DIR=/app/models
      - UPLOAD_DIR=/app/uploads
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
networks:
  traefik:
    external: true

V2 this version is for local usage without expose the ports

compose-v2 .yml

docker network create gnoppixai

services:

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - OLLAMA_KEEP_ALIVE=1h
      - ENABLE_IMAGE_GENERATION=True
      - COMFYUI_BASE_URL=http://stable-diffusion-webui:7860
    networks:
      - ai-stack
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./ollama:/root/.ollama
    ports:
      - "11434:11434" # Add this line to expose the port
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    networks:
      - ai-stack
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - 'OLLAMA_BASE_URL=http://ollama:11434'
      - ENABLE_RAG_WEB_SEARCH=True
      - RAG_WEB_SEARCH_ENGINE=searxng
      - RAG_WEB_SEARCH_RESULT_COUNT=3
      - RAG_WEB_SEARCH_CONCURRENT_REQUESTS=10
      - SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./open-webui:/app/backend/data
    depends_on:
      - ollama
    extra_hosts:
      - host.docker.internal:host-gateway
    ports:
      - "8080:8080" # Add this line to expose the port

  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    networks:
      - ai-stack
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./searxng:/etc/searxng
    depends_on:
      - ollama
      - open-webui
    restart: unless-stopped
    ports:
      - "8081:8080" # Add this line to expose the port

  stable-diffusion-download:
    build: ./stable-diffusion-webui-docker/services/download/
    image: comfy-download
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./stable-diffusion-webui-docker/data:/data

  stable-diffusion-webui:
    build: ./stable-diffusion-webui-docker/services/comfy/
    image: comfy-ui
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - CLI_ARGS=
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./stable-diffusion-webui-docker/data:/data
      - ./stable-diffusion-webui-docker/output:/output
    stop_signal: SIGKILL
    tty: true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [compute, utility]
    restart: unless-stopped
    networks:
      - ai-stack
    ports:
      - "7860:7860" # Add this line to expose the port

  mongo:
    image: mongo
    env_file:
      - .env
    networks:
      - ai-stack
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./whisper/db_data:/data/db
      - ./whisper/db_data/logs/:/var/log/mongodb/
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - MONGO_INITDB_ROOT_USERNAME=${DB_USER:-whishper}
      - MONGO_INITDB_ROOT_PASSWORD=${DB_PASS:-whishper}
    command: ['--logpath', '/var/log/mongodb/mongod.log']
    ports:
      - "27017:27017" # Add this line to expose the port

  translate:
    container_name: whisper-libretranslate
    image: libretranslate/libretranslate:latest-cuda
    env_file:
      - .env
    networks:
      - ai-stack
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./whisper/libretranslate/data:/home/libretranslate/.local/share
      - ./whisper/libretranslate/cache:/home/libretranslate/.local/cache
    user: root
    tty: true
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - LT_DISABLE_WEB_UI=True
      - LT_LOAD_ONLY=${LT_LOAD_ONLY:-en,fr,es}
      - LT_UPDATE_MODELS=True
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
    ports:
      - "5000:5000" # Add this line to expose the port

  whisper:
    container_name: whisper
    pull_policy: always
    image: pluja/whishper:latest-gpu
    env_file:
      - .env
    networks:
      - ai-stack
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./whisper/uploads:/app/uploads
      - ./whisper/logs:/var/log/whishper
      - ./whisper/models:/app/models
    restart: unless-stopped
    depends_on:
      - mongo
      - translate
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - PUBLIC_INTERNAL_API_HOST=${WHISHPER_HOST}
      - PUBLIC_TRANSLATION_API_HOST=${WHISHPER_HOST}
      - PUBLIC_API_HOST=${WHISHPER_HOST:-}
      - PUBLIC_WHISHPER_PROFILE=gpu
      - WHISPER_MODELS_DIR=/app/models
      - UPLOAD_DIR=/app/uploads
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
    ports:
      - "8000:80" # Add this line to expose the port

networks:
  gnoppixai:
    external: true

The missing thing is your LLM’s in case you do not use Gnoppix else apt install gnoppix-llm

Llama 3.1 - censored- under test for chat, wait few days until a dolphin update which unchain the module .

dophin-llama3 - latest uncensored and recommended for chat

lava-llama3 - analyses pictures

llama3-chatqa - document sheet creation and analyses

juggernautXL_v9Rdphoto2Lightning for image creration allinall the best currently

text2voice: WhisperSpeech

If you want try yourself: https://huggingface.co/models?sort=trending