feat(m8): P2 pgvector + Ollama 向量化与语义检索

- PostgreSQL 切换 pgvector/pgvector:pg17 镜像;迁移 000024 建 vector 扩展、
  knowledge_chunks.embedding vector(1024) + HNSW 余弦索引,retrieval_mode 放宽三态
- OllamaEmbedder 本地 bge-m3 批量嵌入,404 惰性 pull 重试,维度/超时校验,可整体关闭
- SemanticRetriever/HybridRetriever + NewRetriever 按 retrieval_mode 分发,缺 embedder 回退 FTS
- 文档入库同步批量向量化;Ollama 故障降级入库 + embedding_failed 事件
- 修复 pgx CopyFrom 对 vector 列二进制编码误读:COPY 基础列后同事务 unnest 批量回填
- 修复降级路径 embeddings=nil 索引越界 panic(Add 与 Reprocess)
- 知识库列表 vectorized_chunk_count + 前端三态检索模式选择与向量化覆盖率
- 单测 embedder/retrievers + 集成 TestKnowledgeVectorLifecycle 全绿

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
ben
2026-08-12 15:16:32 +08:00
parent 6708c226a5
commit b536672000
26 changed files with 966 additions and 60 deletions
+21 -1
View File
@@ -33,6 +33,13 @@ x-gateway-environment: &gateway-environment
S3_REGION: ${S3_REGION:-us-east-1}
S3_USE_SSL: ${S3_USE_SSL:-false}
S3_MAX_FILE_BYTES: ${S3_MAX_FILE_BYTES:-134217728}
# M8 P2:本地 Ollama 向量化。生产可整体关闭(EMBEDDINGS_ENABLED=false)走纯 FTS。
EMBEDDINGS_ENABLED: ${EMBEDDINGS_ENABLED:-true}
OLLAMA_BASE_URL: ${OLLAMA_BASE_URL:-http://ollama:11434}
EMBEDDING_MODEL: ${EMBEDDING_MODEL:-bge-m3}
EMBEDDING_DIM: ${EMBEDDING_DIM:-1024}
EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-64}
EMBEDDING_TIMEOUT: ${EMBEDDING_TIMEOUT:-120s}
x-backend-service: &backend-service
image: ai-gateway-go:${GATEWAY_VERSION:-0.10.0}
@@ -47,8 +54,9 @@ x-backend-service: &backend-service
- no-new-privileges:true
services:
# M8 P2:pgvector 镜像提供 vector 扩展;数据卷与 postgres:17-alpine 兼容。
postgres:
image: postgres:17-alpine
image: pgvector/pgvector:pg17
environment:
POSTGRES_DB: ${POSTGRES_DB:-gateway}
POSTGRES_USER: ${POSTGRES_USER:-gateway}
@@ -116,6 +124,9 @@ services:
condition: service_healthy
minio:
condition: service_started
# 向量化不阻断 API 启动:ollama 未就绪时知识库入库降级。
ollama:
condition: service_started
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8080/readyz"]
interval: 10s
@@ -192,6 +203,14 @@ services:
- minio-data:/data
restart: unless-stopped
# M8 P2:本地向量化(默认 bge-m3)。stateful,首次 embed 惰性 pull 模型;
# 生产不需要时可移除并设 EMBEDDINGS_ENABLED=false 走纯 FTS。
ollama:
image: ollama/ollama:latest
volumes:
- ollama-models:/root/.ollama
restart: unless-stopped
bootstrap-admin:
<<: *backend-service
profiles: ["tools"]
@@ -209,3 +228,4 @@ volumes:
postgres-data:
redis-critical-data:
minio-data:
ollama-models: