# Production deployment

This bundle builds the Go services and both Art Design Pro applications from
source. PostgreSQL (with pgvector), two Redis roles, MinIO (object storage, M8)
and a local Ollama container (vectorization, M8) are included; ClickHouse is not
required. Neither MinIO nor Ollama is a startup dependency: the gateway warns
and refuses file uploads until the bucket is reachable, and knowledge-base
documents are still stored (with embedding set to NULL) when Ollama is down,
with retrieval falling back to full-text search.
The scheduler is a separate stateless worker backed by PostgreSQL leases; run at
least one `scheduler-worker` replica whenever scheduled tasks are enabled.

## Prerequisites

- Docker Engine with Compose v2
- At least 4 CPU cores, 8 GiB RAM and 30 GiB free disk for an initial build
- An external TLS reverse proxy or load balancer
- A backup destination for the PostgreSQL volume

## First deployment

Run all commands from the repository root:

```bash
cp deploy/production.env.example deploy/production.env
chmod 600 deploy/production.env
# Edit deploy/production.env and replace every CHANGE_ME value.

docker compose \
  --env-file deploy/production.env \
  -f deploy/docker-compose.production.yml \
  config --quiet

docker compose \
  --env-file deploy/production.env \
  -f deploy/docker-compose.production.yml \
  up -d --build
```

Create the initial administrator once:

```bash
docker compose \
  --env-file deploy/production.env \
  -f deploy/docker-compose.production.yml \
  --profile tools run --rm bootstrap-admin
```

Then remove `BOOTSTRAP_ADMIN_PASSWORD` from `deploy/production.env` and use the
admin UI to create database-backed gateway API keys.

## Endpoints

- API and OpenAI-compatible gateway: `127.0.0.1:8080`
- Admin UI: `http://127.0.0.1:8081/admin/`
- Portal UI: `http://127.0.0.1:8082/portal/`
- Liveness/readiness: `/healthz` and `/readyz`

Ports bind to loopback by default. Terminate TLS at a reverse proxy and forward
to these endpoints. Change `*_BIND_IP` only when the host firewall and network
policy are already in place.

## Operations

Check status and logs:

```bash
docker compose --env-file deploy/production.env -f deploy/docker-compose.production.yml ps
docker compose --env-file deploy/production.env -f deploy/docker-compose.production.yml logs --tail=200 gateway-api
docker compose --env-file deploy/production.env -f deploy/docker-compose.production.yml logs --tail=200 scheduler-worker
curl --fail http://127.0.0.1:8080/readyz
```

For upgrades, back up PostgreSQL first, change `GATEWAY_VERSION`, then run the
same `up -d --build` command. The one-shot migrator applies forward migrations
before the API starts. Do not use `docker compose down -v` in production because
it removes persistent data.

The bundled database URLs use `sslmode=disable` only for the private Compose
network. When using an external PostgreSQL or Redis service, require TLS and use
`sslmode=verify-full` / `rediss://` as supported by that service.

### Scheduled tasks

- `scheduler-worker` calls the gateway over `SCHEDULER_GATEWAY_BASE_URL`; keep it
  on the private application network and do not expose a scheduler port.
- Task API keys are encrypted with `CREDENTIAL_MASTER_KEY`. Back up the active
  key and its historical keyring together with PostgreSQL before rotation.
- `SCHEDULER_MAX_ATTEMPTS` controls both ordinary execution retries and stale
  lease recovery. Final success/failure is published through outbox and can be
  delivered to the selected notification channel and the creator's inbox.
- Multiple replicas are safe because due tasks and runs are claimed with
  PostgreSQL row locks and `SKIP LOCKED`.

### Vectorization and object storage

- The PostgreSQL image is `pgvector/pgvector:pg17` (data-volume compatible with
  `postgres:17-alpine`); migration `000024` creates the `vector` extension and
  adds the HNSW embedding column. `EMBEDDING_DIM` must stay at `1024` to match
  the `vector(1024)` column.
- Ollama runs locally and lazily pulls `bge-m3` (~1.2 GiB) on first embedding
  request. Set `EMBEDDINGS_ENABLED=false` to disable vectorization entirely.
- Back up the `minio-data` and `ollama-models` volumes alongside PostgreSQL.
- If you previously deployed with `postgres:17-alpine`, back up the PostgreSQL
  volume before switching images.

### CREDENTIAL_MASTER_KEY 持久化（重要）

- 所有加密凭据（Provider API Key、TOTP 密钥、Webhook 签名、调度任务 Key、
  应用运行时 Key）都用 `CREDENTIAL_MASTER_KEY` 加密。**该密钥必须持久化**：
  每次部署换新 key 会让全部已存凭据无法解密。
- 本地 compose 首次启动前执行
  `openssl rand -base64 32 > deploy/.env`（compose 自动读取该文件），
  之后重启/重建复用同一 key；`.gitignore` 已排除 `deploy/.env`。
- 需要轮换时使用管理端「供应商 → 凭据轮换」接口（同一 KEK 版本内重加密），
  并保留历史 keyring；不要直接更换 `CREDENTIAL_MASTER_KEY` 值。
- 若误换 key：管理端 Provider 列表会降级显示「凭据无法解密」警示（不会
  让整个页面报错），需重新保存各 Provider 的 API Key 恢复。

### 多架构发布(ARM64)

- 后端镜像支持 amd64 + arm64 交叉编译(CGO_ENABLED=0):
  `./scripts/publish-images.sh registry.example.com/ai-gateway:0.11.0`
  (需先 `docker buildx create --use`)。
- 前端镜像(node/nginx)由 Docker Hub 提供多架构基础镜像,可直接 buildx 构建。
- ARM64 节点部署:arm64 设备上 `docker compose up -d` 即可拉取 arm64 变体。