svc-observe 📊

Visibility into every token — LLM observability, trace analytics, and continuous evaluation for the Generate One platform. Every model call, measured.

---

✨ Overview

svc-observe runs Langfuse v3 — a comprehensive LLM observability platform that captures every model request flowing through LiteLLM. Traces are scored, evaluated, and stored for quality analysis, cost tracking, and debugging. The stack is fully self-contained: ClickHouse provides columnar analytics for trace data, MinIO offers S3-compatible blob storage for events and media, and PostgreSQL (from g1-core) stores project and user metadata.

Six automated evaluators run at 15% sampling rate using g1-llm-micro and g1-llm-turbo to continuously assess output quality — without creating evaluation loops (tag-filtered).

---

🏗️ Architecture

graph TD
    subgraph Producers ["Trace Producers"]
        LL["g1-llm / LiteLLM\nAll model requests"]
        GR["g1-brain / Graphiti\nLLM calls via LiteLLM"]
    end

    subgraph svc-observe ["svc-observe (svc-observe)"]
        LW["langfuse-web\n:3000\nUI · Public API · Ingestion"]
        WK["langfuse-worker\nBackground jobs\nEvals · Exports · Ingestion"]
        CH["ClickHouse\n:8123 (HTTP)\n:9000 (native)"]
        MN["MinIO\n:9000 (API)\n:9001 (console)"]
    end

    subgraph Deps ["External Dependencies"]
        PG["g1-core / PostgreSQL\nlangfuse DB"]
        VK["g1-core / Valkey\nDB 8"]
        AK["svc-auth / Authentik\nOIDC SSO"]
    end

    LL & GR -->|"SDK traces"| LW
    LW --> PG
    LW --> CH
    LW --> MN
    LW --- WK
    WK --> CH & MN & PG
    VK --> LW & WK
    AK -->|"OIDC SSO"| LW
    User -->|"https://observe.generate.one"| LW

---

📦 Services

Service	Image	Port	Description
langfuse-web	langfuse/langfuse:3	3000	Web UI + API server — trace ingestion, dashboard, evaluations
langfuse-worker	langfuse/langfuse-worker:3	—	Background processor: evaluations, exports, async ingestion
clickhouse	clickhouse/clickhouse-server	8123 (HTTP), 9000 (native)	Columnar analytics database for all trace and event data
minio	cgr.dev/chainguard/minio	9000 (API), 9001 (console)	S3-compatible blob storage for events and media

Persistent Volumes

Volume	Purpose
clickhouse-data	ClickHouse data files
clickhouse-logs	ClickHouse server logs
minio-data	MinIO object storage

---

🚀 Quick Start

# Service directory
cd /data/coolify/services/f4o0gg0oocoog4ggo848oswc

# Apply changes
docker compose up -d

# View logs
docker compose logs -f langfuse-web
docker compose logs -f langfuse-worker

# Check public API health
curl -s https://observe.generate.one/api/public/health

# Check ClickHouse
docker exec -it <clickhouse-container> clickhouse-client --user clickhouse --password $CLICKHOUSE_PASSWORD

Key Endpoints

Endpoint	Purpose
https://observe.generate.one/	Langfuse dashboard
https://observe.generate.one/api/public/	Public API (trace ingestion)
http://langfuse-web-f4o0gg0oocoog4ggo848oswc:3000	Internal cross-stack endpoint

---

🔧 Configuration

Variable	Description	Notes
NEXTAUTH_SECRET	NextAuth session encryption key	Required
NEXTAUTH_URL	Public URL	https://observe.generate.one
DATABASE_URL	PostgreSQL connection string	langfuse database on g1-core
LANGFUSE_SALT	Encryption salt	Must never change
ENCRYPTION_KEY	Data encryption key	Must never change
CLICKHOUSE_PASSWORD	ClickHouse auth
MINIO_ROOT_PASSWORD	MinIO root password	Also used for S3 bucket access
VALKEY_HOST	Valkey host	valkey-u8kg4sscgcccok088048so4
VALKEY_PASSWORD	Valkey auth	DB index 8
AUTH_AUTHENTIK_CLIENT_ID	Authentik OIDC client ID	Application slug: langfuse
AUTH_AUTHENTIK_CLIENT_SECRET	Authentik OIDC client secret
TELEMETRY_ENABLED	Langfuse telemetry	false
CLICKHOUSE_CLUSTER_ENABLED	ClickHouse cluster mode	false (single node)

S3 Storage Configuration

MinIO uses path-style addressing for both event and media uploads:

Bucket	Prefix	Purpose
langfuse	events/	Trace event storage
langfuse	media/	Media upload storage

---

📈 Evaluators

6 automated evaluators running at 15% sampling rate using g1-llm-micro and g1-llm-turbo model tiers. Tag filter langfuse-eval prevents evaluation loops.

---

🔗 Dependencies

Depends On

• g1-core — PostgreSQL (langfuse database), Valkey (DB 8)
• svc-auth — Authentik OIDC for SSO login

Depended On By

• g1-llm — LiteLLM sends all traces to Langfuse
• g1-brain — Graphiti LLM calls traced via LiteLLM

Note: ClickHouse and MinIO are self-contained within this stack — they do not use g1-core.

---

🗂️ Quick Reference

Item	Value
Domain	observe.generate.one
Coolify Project	svc-observe
Coolify Service UUID	f4o0gg0oocoog4ggo848oswc
Docker Network	coolify (external) + default (internal stack)
Langfuse Version	v3
OIDC Issuer	https://auth.generate.one/application/o/langfuse

---

🛡️ Part of Generate One

Generate One — AI infrastructure that answers to you.

Sovereignty First · Composable · Open by Default · Developer Empathy

Self-hosted, sovereign AI platform. generate.one