reflector/docsv2/custom-ca-setup.md

Custom CA Certificate Setup

Use a private Certificate Authority (CA) with Reflector self-hosted deployments. This covers two scenarios:

1. Custom local domain — Serve Reflector over HTTPS on an internal domain (e.g., reflector.local) using certs signed by your own CA
2. Backend CA trust — Let Reflector's backend services (server, workers, GPU) make HTTPS calls to GPU, LLM, or other internal services behind your private CA

Both can be used independently or together.

Quick Start

Generate test certificates

./scripts/generate-certs.sh reflector.local

This creates certs/ with:

• ca.key + ca.crt — Root CA (10-year validity)
• server-key.pem + server.pem — Server certificate (1-year, SAN: domain + localhost + 127.0.0.1)

Deploy with custom CA + domain

# Add domain to /etc/hosts on the server (use 127.0.0.1 for local, or server LAN IP for network access)
echo "127.0.0.1 reflector.local" | sudo tee -a /etc/hosts

# Run setup — pass the certs directory
./scripts/setup-selfhosted.sh --gpu --caddy --domain reflector.local --custom-ca certs/

# Trust the CA on your machine (see "Trust the CA" section below)

Deploy with CA trust only (GPU/LLM behind private CA)

# Only need the CA cert file — no Caddy TLS certs needed
./scripts/setup-selfhosted.sh --hosted --custom-ca /path/to/corporate-ca.crt

How --custom-ca Works

The flag accepts a directory or a single file:

Directory mode

--custom-ca certs/

Looks for these files by convention:

• ca.crt (required) — CA certificate to trust
• server.pem + server-key.pem (optional) — TLS certificate/key for Caddy

If server.pem + server-key.pem are found AND --domain is provided:

• Caddy serves HTTPS using those certs
• Backend containers trust the CA for outbound calls

If only ca.crt is found:

• Backend containers trust the CA for outbound calls
• Caddy is unaffected (uses Let's Encrypt, self-signed, or no Caddy)

Single file mode

--custom-ca /path/to/corporate-ca.crt

Only injects CA trust into backend containers. No Caddy TLS changes.

Scenarios

Scenario 1: Custom local domain

Your Reflector instance runs on an internal network. You want https://reflector.local with proper TLS (no browser warnings).

# 1. Generate certs
./scripts/generate-certs.sh reflector.local

# 2. Add to /etc/hosts on the server
echo "127.0.0.1 reflector.local" | sudo tee -a /etc/hosts

# 3. Deploy
./scripts/setup-selfhosted.sh --gpu --garage --caddy --domain reflector.local --custom-ca certs/

# 4. Trust the CA on your machine (see "Trust the CA" section below)

If other machines on the network need to access it, add the server's LAN IP to /etc/hosts on those machines instead:

echo "192.168.1.100 reflector.local" | sudo tee -a /etc/hosts

And include that IP as an extra SAN when generating certs:

./scripts/generate-certs.sh reflector.local "IP:192.168.1.100"

Scenario 2: GPU/LLM behind corporate CA

Your GPU or LLM server (e.g., https://gpu.internal.corp) uses certificates signed by your corporate CA. Reflector's backend needs to trust that CA for outbound HTTPS calls.

# Get the CA certificate from your IT team (PEM format)
# Then deploy — Caddy can still use Let's Encrypt or self-signed
./scripts/setup-selfhosted.sh --hosted --garage --caddy --custom-ca /path/to/corporate-ca.crt

This works because:

• TLS cert/key = "this is my identity" — for Caddy to serve HTTPS to browsers
• CA cert = "I trust this authority" — for backend containers to verify outbound connections

Your Reflector frontend can use Let's Encrypt (public domain) or self-signed certs, while the backend trusts a completely different CA for GPU/LLM calls.

Scenario 3: Both combined (same CA)

Custom domain + GPU/LLM all behind the same CA:

./scripts/generate-certs.sh reflector.local "DNS:gpu.local"
./scripts/setup-selfhosted.sh --gpu --garage --caddy --domain reflector.local --custom-ca certs/

Scenario 4: Multiple CAs (local domain + remote GPU on different CA)

Your Reflector uses one CA for reflector.local, but the GPU host uses a different CA:

# Your local domain setup
./scripts/generate-certs.sh reflector.local

# Deploy with your CA + trust the GPU host's CA too
./scripts/setup-selfhosted.sh --hosted --garage --caddy \
    --domain reflector.local \
    --custom-ca certs/ \
    --extra-ca /path/to/gpu-machine-ca.crt

--extra-ca appends additional CA certs to the trust bundle. Backend containers trust ALL CAs — your local domain AND the GPU host's certs both work.

You can repeat --extra-ca for multiple remote services:

--extra-ca /path/to/gpu-ca.crt --extra-ca /path/to/llm-ca.crt

For setting up a dedicated GPU host, see Standalone GPU Host Setup.

Trust the CA on Client Machines

After deploying, clients need to trust the CA to avoid browser warnings.

macOS

sudo security add-trusted-cert -d -r trustRoot \
    -k /Library/Keychains/System.keychain certs/ca.crt

Linux (Ubuntu/Debian)

sudo cp certs/ca.crt /usr/local/share/ca-certificates/reflector-ca.crt
sudo update-ca-certificates

Linux (RHEL/Fedora)

sudo cp certs/ca.crt /etc/pki/ca-trust/source/anchors/reflector-ca.crt
sudo update-ca-trust

Windows (PowerShell as admin)

Import-Certificate -FilePath .\certs\ca.crt -CertStoreLocation Cert:\LocalMachine\Root

Firefox (all platforms)

Firefox uses its own certificate store:

1. Settings > Privacy & Security > View Certificates
2. Authorities tab > Import
3. Select ca.crt and check "Trust this CA to identify websites"

How It Works Internally

Docker entrypoint CA injection

Each backend container (server, worker, beat, hatchet workers, GPU) has an entrypoint script (docker-entrypoint.sh) that:

1. Checks if a CA cert is mounted at /usr/local/share/ca-certificates/custom-ca.crt
2. If present, runs update-ca-certificates to create a combined bundle (system CAs + custom CA)
3. Sets environment variables so all Python/gRPC libraries use the combined bundle:

Env var	Covers
SSL_CERT_FILE	httpx, OpenAI SDK, llama-index, Python ssl module
REQUESTS_CA_BUNDLE	requests library (transitive dependencies)
CURL_CA_BUNDLE	curl CLI (container healthchecks)
GRPC_DEFAULT_SSL_ROOTS_FILE_PATH	grpcio (Hatchet gRPC client)

When no CA cert is mounted, the entrypoint is a no-op — containers behave exactly as before.

Why this replaces manual certifi patching

Previously, the workaround for trusting a private CA in Python was to patch certifi's bundle directly:

# OLD approach — fragile, do NOT use
cat custom-ca.crt >> $(python -c "import certifi; print(certifi.where())")

This breaks whenever certifi is updated (any pip install/uv sync overwrites the bundle and the CA is lost).

Our entrypoint approach is permanent because:

1. SSL_CERT_FILE is checked by Python's ssl.create_default_context() before falling back to certifi.where(). When set, certifi's bundle is never read.
2. REQUESTS_CA_BUNDLE similarly overrides certifi for the requests library.
3. The CA is injected at container startup (runtime), not baked into the Python environment. It survives image rebuilds, dependency updates, and uv sync.

Python SSL lookup chain:
  ssl.create_default_context()
    → SSL_CERT_FILE env var? → YES → use combined bundle (system + custom CA) ✓
    → (certifi.where() is never reached)

This covers all outbound HTTPS calls: httpx (transcription, diarization, translation, webhooks), OpenAI SDK (transcription), llama-index (LLM/summarization), and requests (transitive dependencies).

Compose override

The setup script generates docker-compose.ca.yml which mounts the CA cert into every backend container as a read-only bind mount. This file is:

• Only generated when --custom-ca is passed
• Deleted on re-runs without --custom-ca (prevents stale overrides)
• Added to .gitignore

Node.js (frontend)

The web container uses NODE_EXTRA_CA_CERTS which adds to Node's trust store (unlike Python's SSL_CERT_FILE which replaces it). This is set via the compose override.

Generate Your Own CA (Manual)

If you prefer not to use generate-certs.sh:

# 1. Create CA
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
    -out ca.crt -subj "/CN=My CA/O=My Organization"

# 2. Create server key
openssl genrsa -out server-key.pem 2048

# 3. Create CSR with SANs
openssl req -new -key server-key.pem -out server.csr \
    -subj "/CN=reflector.local" \
    -addext "subjectAltName=DNS:reflector.local,DNS:localhost,IP:127.0.0.1"

# 4. Sign with CA
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -out server.pem -days 365 -sha256 \
    -copy_extensions copyall

# 5. Clean up
rm server.csr ca.srl

Using Existing Corporate Certificates

If your organization already has a CA:

1. Get the CA certificate in PEM format from your IT team
2. If you have a PKCS#12 (.p12/.pfx) bundle, extract the CA cert:

   openssl pkcs12 -in bundle.p12 -cacerts -nokeys -out ca.crt
   

3. If you have multiple intermediate CAs, concatenate them into one PEM file:

   cat intermediate-ca.crt root-ca.crt > ca.crt
   

Troubleshooting

Browser: "Your connection is not private"

The CA is not trusted on the client machine. See "Trust the CA" section above.

Check certificate expiry:

openssl x509 -noout -dates -in certs/server.pem

Backend: SSL: CERTIFICATE_VERIFY_FAILED

CA cert not mounted or not loaded. Check inside the container:

docker compose exec server env | grep SSL_CERT_FILE
docker compose exec server python -c "
import ssl, os
print('SSL_CERT_FILE:', os.environ.get('SSL_CERT_FILE', 'not set'))
ctx = ssl.create_default_context()
print('CA certs loaded:', ctx.cert_store_stats())
"

Caddy: "certificate is not valid for any names"

Domain in Caddyfile doesn't match the certificate's SAN/CN. Check:

openssl x509 -noout -text -in certs/server.pem | grep -A1 "Subject Alternative Name"

Certificate chain issues

If you have intermediate CAs, concatenate them into server.pem:

cat server-cert.pem intermediate-ca.pem > certs/server.pem

Verify the chain:

openssl verify -CAfile certs/ca.crt certs/server.pem

Certificate renewal

Custom CA certs are NOT auto-renewed (unlike Let's Encrypt). Replace cert files and restart:

# Replace certs
cp new-server.pem certs/server.pem
cp new-server-key.pem certs/server-key.pem

# Restart Caddy to pick up new certs
docker compose restart caddy