mirror of
https://github.com/Monadical-SAS/reflector.git
synced 2025-12-21 12:49:06 +00:00
install from scratch docs
This commit is contained in:
Igor Loskutov
253
docs/docs/installation/auth-setup.md
Normal file
253
docs/docs/installation/auth-setup.md
Normal file
@@ -0,0 +1,253 @@
|
||||
---
|
||||
sidebar_position: 5
|
||||
title: Authentication Setup
|
||||
---
|
||||
|
||||
# Authentication Setup
|
||||
|
||||
This page covers authentication setup in detail. For the complete deployment guide, see [Deployment Guide](./overview).
|
||||
|
||||
Reflector uses [Authentik](https://goauthentik.io/) for OAuth/OIDC authentication. This guide walks you through setting up Authentik and connecting it to Reflector.
|
||||
|
||||
## Overview
|
||||
|
||||
Reflector's authentication flow:
|
||||
1. User clicks "Sign In" on frontend
|
||||
2. Frontend redirects to Authentik login page
|
||||
3. User authenticates with Authentik
|
||||
4. Authentik redirects back with OAuth tokens
|
||||
5. Frontend stores tokens, backend verifies JWT signature
|
||||
|
||||
## Option 1: Self-Hosted Authentik (Same Server)
|
||||
|
||||
This setup runs Authentik on the same server as Reflector, with Caddy proxying to both.
|
||||
|
||||
### Step 1: Deploy Authentik
|
||||
|
||||
```bash
|
||||
# Create directory for Authentik
|
||||
mkdir -p ~/authentik && cd ~/authentik
|
||||
|
||||
# Download docker-compose file
|
||||
curl -O https://goauthentik.io/docker-compose.yml
|
||||
|
||||
# Generate secrets and bootstrap credentials
|
||||
cat > .env << 'EOF'
|
||||
PG_PASS=$(openssl rand -base64 36 | tr -d '\n')
|
||||
AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60 | tr -d '\n')
|
||||
AUTHENTIK_ERROR_REPORTING__ENABLED=false
|
||||
AUTHENTIK_BOOTSTRAP_PASSWORD=YourSecurePassword123
|
||||
AUTHENTIK_BOOTSTRAP_EMAIL=admin@example.com
|
||||
EOF
|
||||
|
||||
# Start Authentik
|
||||
sudo docker compose up -d
|
||||
```
|
||||
|
||||
Authentik takes ~2 minutes to run migrations and apply blueprints on first start.
|
||||
|
||||
### Step 2: Connect Authentik to Reflector's Network
|
||||
|
||||
Since Authentik runs in a separate Docker Compose project, connect it to Reflector's network so Caddy can proxy to it:
|
||||
|
||||
```bash
|
||||
# Wait for Authentik to be healthy
|
||||
sleep 120
|
||||
|
||||
# Connect Authentik server to Reflector's network
|
||||
sudo docker network connect reflector_default authentik-server-1
|
||||
```
|
||||
|
||||
**Important:** This step must be repeated if you restart Authentik with `docker compose down`. Add it to your deployment scripts or use `docker compose up -d` (which preserves containers) instead of down/up.
|
||||
|
||||
### Step 3: Add Authentik to Caddy
|
||||
|
||||
Edit your `Caddyfile` to add the Authentik domain:
|
||||
|
||||
```
|
||||
app.example.com {
|
||||
reverse_proxy web:3000
|
||||
}
|
||||
|
||||
api.example.com {
|
||||
reverse_proxy server:1250
|
||||
}
|
||||
|
||||
authentik.example.com {
|
||||
reverse_proxy authentik-server-1:9000
|
||||
}
|
||||
```
|
||||
|
||||
Reload Caddy:
|
||||
```bash
|
||||
sudo docker exec reflector-caddy-1 caddy reload --config /etc/caddy/Caddyfile
|
||||
```
|
||||
|
||||
### Step 4: Create OAuth2 Provider in Authentik
|
||||
|
||||
1. **Login to Authentik Admin** at `https://authentik.example.com/`
|
||||
- Username: `akadmin`
|
||||
- Password: The `AUTHENTIK_BOOTSTRAP_PASSWORD` you set in .env
|
||||
|
||||
2. **Create OAuth2 Provider:**
|
||||
- Go to **Applications > Providers > Create**
|
||||
- Select **OAuth2/OpenID Provider**
|
||||
- Configure:
|
||||
- **Name**: `Reflector`
|
||||
- **Authorization flow**: `default-provider-authorization-implicit-consent`
|
||||
- **Client type**: `Confidential`
|
||||
- **Client ID**: Note this value (auto-generated)
|
||||
- **Client Secret**: Note this value (auto-generated)
|
||||
- **Redirect URIs**: Add entry with:
|
||||
```
|
||||
https://app.example.com/api/auth/callback/authentik
|
||||
```
|
||||
- Click **Finish**
|
||||
|
||||
3. **Create Application:**
|
||||
- Go to **Applications > Applications > Create**
|
||||
- Configure:
|
||||
- **Name**: `Reflector`
|
||||
- **Slug**: `reflector` (auto-filled)
|
||||
- **Provider**: Select the `Reflector` provider you just created
|
||||
- Click **Create**
|
||||
|
||||
### Step 5: Get Public Key for JWT Verification
|
||||
|
||||
Extract the public key from Authentik's JWKS endpoint:
|
||||
|
||||
```bash
|
||||
curl -s https://authentik.example.com/application/o/reflector/jwks/ | \
|
||||
jq -r '.keys[0].x5c[0]' | base64 -d | openssl x509 -pubkey -noout \
|
||||
> ~/reflector/server/reflector/auth/jwt/keys/authentik_public.pem
|
||||
```
|
||||
|
||||
### Step 6: Update docker-compose.prod.yml
|
||||
|
||||
Add a volume mount for the JWT keys directory to the server and worker services:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
image: monadicalsas/reflector-backend:latest
|
||||
# ... other config ...
|
||||
volumes:
|
||||
- server_data:/app/data
|
||||
- ./server/reflector/auth/jwt/keys:/app/reflector/auth/jwt/keys:ro
|
||||
|
||||
worker:
|
||||
image: monadicalsas/reflector-backend:latest
|
||||
# ... other config ...
|
||||
volumes:
|
||||
- server_data:/app/data
|
||||
- ./server/reflector/auth/jwt/keys:/app/reflector/auth/jwt/keys:ro
|
||||
```
|
||||
|
||||
### Step 7: Configure Reflector Backend
|
||||
|
||||
Update `server/.env`:
|
||||
```env
|
||||
# Authentication
|
||||
AUTH_BACKEND=jwt
|
||||
AUTH_JWT_PUBLIC_KEY=authentik_public.pem
|
||||
AUTH_JWT_AUDIENCE=<your-client-id>
|
||||
CORS_ALLOW_CREDENTIALS=true
|
||||
```
|
||||
|
||||
Replace `<your-client-id>` with the Client ID from Step 4.
|
||||
|
||||
### Step 8: Configure Reflector Frontend
|
||||
|
||||
Update `www/.env`:
|
||||
```env
|
||||
# Authentication
|
||||
FEATURE_REQUIRE_LOGIN=true
|
||||
|
||||
# Authentik OAuth
|
||||
AUTHENTIK_ISSUER=https://authentik.example.com/application/o/reflector
|
||||
AUTHENTIK_REFRESH_TOKEN_URL=https://authentik.example.com/application/o/token/
|
||||
AUTHENTIK_CLIENT_ID=<your-client-id>
|
||||
AUTHENTIK_CLIENT_SECRET=<your-client-secret>
|
||||
|
||||
# NextAuth
|
||||
NEXTAUTH_SECRET=<generate-with-openssl-rand-hex-32>
|
||||
```
|
||||
|
||||
### Step 9: Restart Services
|
||||
|
||||
```bash
|
||||
cd ~/reflector
|
||||
sudo docker compose -f docker-compose.prod.yml up -d --force-recreate server worker web
|
||||
```
|
||||
|
||||
### Step 10: Verify Authentication
|
||||
|
||||
1. Visit `https://app.example.com`
|
||||
2. Click "Log in" or navigate to `/api/auth/signin`
|
||||
3. Click "Sign in with Authentik"
|
||||
4. Login with your Authentik credentials
|
||||
5. You should be redirected back and see "Log out" in the header
|
||||
|
||||
## Option 2: Disable Authentication
|
||||
|
||||
For testing or internal deployments where authentication isn't needed:
|
||||
|
||||
**Backend `server/.env`:**
|
||||
```env
|
||||
AUTH_BACKEND=none
|
||||
```
|
||||
|
||||
**Frontend `www/.env`:**
|
||||
```env
|
||||
FEATURE_REQUIRE_LOGIN=false
|
||||
```
|
||||
|
||||
**Note:** The pre-built Docker images have `FEATURE_REQUIRE_LOGIN=true` baked in. To disable auth, you'll need to rebuild the frontend image with the env var set at build time, or set up Authentik.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Invalid redirect URI" error
|
||||
- Verify the redirect URI in Authentik matches exactly:
|
||||
```
|
||||
https://app.example.com/api/auth/callback/authentik
|
||||
```
|
||||
- Check for trailing slashes - they must match exactly
|
||||
|
||||
### "Invalid audience" JWT error
|
||||
- Ensure `AUTH_JWT_AUDIENCE` in `server/.env` matches the Client ID from Authentik
|
||||
- The audience value is the OAuth Client ID, not the issuer URL
|
||||
|
||||
### "JWT verification failed" error
|
||||
- Verify the public key file is mounted in the container
|
||||
- Check `AUTH_JWT_PUBLIC_KEY` points to the correct filename
|
||||
- Ensure the key was extracted from the correct provider's JWKS endpoint
|
||||
|
||||
### Caddy returns 503 for Authentik
|
||||
- Verify Authentik container is connected to Reflector's network:
|
||||
```bash
|
||||
sudo docker network connect reflector_default authentik-server-1
|
||||
```
|
||||
- Check Authentik is healthy: `cd ~/authentik && sudo docker compose ps`
|
||||
|
||||
### Users can't access protected pages
|
||||
- Verify `FEATURE_REQUIRE_LOGIN=true` in frontend
|
||||
- Check `AUTH_BACKEND=jwt` in backend
|
||||
- Verify CORS settings allow credentials
|
||||
|
||||
### Token refresh errors
|
||||
- Ensure Redis is running (frontend uses Redis for token caching)
|
||||
- Verify `KV_URL` is set correctly in frontend env
|
||||
- Check `AUTHENTIK_REFRESH_TOKEN_URL` is correct
|
||||
|
||||
## API Key Authentication
|
||||
|
||||
For programmatic access (scripts, integrations), users can generate API keys:
|
||||
|
||||
1. Login to Reflector
|
||||
2. Go to Settings > API Keys
|
||||
3. Click "Generate New Key"
|
||||
4. Use the key in requests:
|
||||
```bash
|
||||
curl -H "X-API-Key: your-api-key" https://api.example.com/v1/transcripts
|
||||
```
|
||||
|
||||
API keys are stored hashed and can be revoked at any time.
|
||||
@@ -1,23 +1,188 @@
|
||||
---
|
||||
sidebar_position: 3
|
||||
title: Docker Deployment
|
||||
title: Docker Reference
|
||||
---
|
||||
|
||||
# Docker Deployment
|
||||
# Docker Reference
|
||||
|
||||
See the [Docker directory](https://github.com/monadical-sas/reflector/tree/main/docker) in the repository for the complete Docker deployment configuration.
|
||||
This page documents the Docker Compose configuration for Reflector. For the complete deployment guide, see [Deployment Guide](./overview).
|
||||
|
||||
## Quick Start
|
||||
## Services
|
||||
|
||||
1. Clone the repository
|
||||
2. Navigate to `/docker` directory
|
||||
3. Copy `.env.example` to `.env`
|
||||
4. Configure environment variables
|
||||
5. Run `docker compose up -d`
|
||||
The `docker-compose.prod.yml` includes these services:
|
||||
|
||||
## Configuration
|
||||
| Service | Image | Purpose |
|
||||
|---------|-------|---------|
|
||||
| `web` | `monadicalsas/reflector-frontend` | Next.js frontend |
|
||||
| `server` | `monadicalsas/reflector-backend` | FastAPI backend |
|
||||
| `worker` | `monadicalsas/reflector-backend` | Celery worker for background tasks |
|
||||
| `beat` | `monadicalsas/reflector-backend` | Celery beat scheduler |
|
||||
| `redis` | `redis:7.2-alpine` | Message broker and cache |
|
||||
| `postgres` | `postgres:17-alpine` | Primary database |
|
||||
| `caddy` | `caddy:2-alpine` | Reverse proxy with auto-SSL |
|
||||
|
||||
Check the repository for:
|
||||
- `docker-compose.yml` - Service definitions
|
||||
- `.env.example` - Environment variables
|
||||
- `Caddyfile` - Reverse proxy configuration
|
||||
## Environment Files
|
||||
|
||||
Reflector uses two separate environment files:
|
||||
|
||||
### Backend (`server/.env`)
|
||||
|
||||
Used by: `server`, `worker`, `beat`
|
||||
|
||||
Key variables:
|
||||
```env
|
||||
# Database connection
|
||||
DATABASE_URL=postgresql+asyncpg://reflector:reflector@postgres:5432/reflector
|
||||
|
||||
# Redis
|
||||
REDIS_HOST=redis
|
||||
CELERY_BROKER_URL=redis://redis:6379/1
|
||||
CELERY_RESULT_BACKEND=redis://redis:6379/1
|
||||
|
||||
# API domain and CORS
|
||||
BASE_URL=https://api.example.com
|
||||
CORS_ORIGIN=https://app.example.com
|
||||
|
||||
# Modal GPU processing
|
||||
TRANSCRIPT_BACKEND=modal
|
||||
TRANSCRIPT_URL=https://...
|
||||
TRANSCRIPT_MODAL_API_KEY=...
|
||||
```
|
||||
|
||||
### Frontend (`www/.env`)
|
||||
|
||||
Used by: `web`
|
||||
|
||||
Key variables:
|
||||
```env
|
||||
# Domain configuration
|
||||
SITE_URL=https://app.example.com
|
||||
API_URL=https://api.example.com
|
||||
WEBSOCKET_URL=wss://api.example.com
|
||||
SERVER_API_URL=http://server:1250
|
||||
|
||||
# Authentication
|
||||
NEXTAUTH_URL=https://app.example.com
|
||||
NEXTAUTH_SECRET=...
|
||||
```
|
||||
|
||||
Note: `API_URL` is used client-side (browser), `SERVER_API_URL` is used server-side (SSR).
|
||||
|
||||
## Volumes
|
||||
|
||||
| Volume | Purpose |
|
||||
|--------|---------|
|
||||
| `redis_data` | Redis persistence |
|
||||
| `postgres_data` | PostgreSQL data |
|
||||
| `server_data` | Uploaded files, local storage |
|
||||
| `caddy_data` | SSL certificates |
|
||||
| `caddy_config` | Caddy configuration |
|
||||
|
||||
## Network
|
||||
|
||||
All services share the default network. The network is marked `attachable: true` to allow external containers (like Authentik) to join.
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Start all services
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### View logs
|
||||
```bash
|
||||
# All services
|
||||
docker compose -f docker-compose.prod.yml logs -f
|
||||
|
||||
# Specific service
|
||||
docker compose -f docker-compose.prod.yml logs server --tail 50
|
||||
```
|
||||
|
||||
### Restart a service
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml restart server
|
||||
```
|
||||
|
||||
### Run database migrations
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml exec server uv run alembic upgrade head
|
||||
```
|
||||
|
||||
### Access database
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml exec postgres psql -U reflector
|
||||
```
|
||||
|
||||
### Pull latest images
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml pull
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### Stop all services
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml down
|
||||
```
|
||||
|
||||
### Full reset (WARNING: deletes data)
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml down -v
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
### Using a different database
|
||||
|
||||
To use an external PostgreSQL:
|
||||
|
||||
1. Remove `postgres` service from compose file
|
||||
2. Update `DATABASE_URL` in `server/.env`:
|
||||
```env
|
||||
DATABASE_URL=postgresql+asyncpg://user:pass@external-host:5432/reflector
|
||||
```
|
||||
|
||||
### Using external Redis
|
||||
|
||||
1. Remove `redis` service from compose file
|
||||
2. Update Redis settings in `server/.env`:
|
||||
```env
|
||||
REDIS_HOST=external-redis-host
|
||||
CELERY_BROKER_URL=redis://external-redis-host:6379/1
|
||||
```
|
||||
|
||||
### Adding Authentik
|
||||
|
||||
To add Authentik for authentication, see [Authentication Setup](./auth-setup). Quick steps:
|
||||
|
||||
1. Deploy Authentik separately
|
||||
2. Connect to Reflector's network:
|
||||
```bash
|
||||
docker network connect reflector_default authentik-server-1
|
||||
```
|
||||
3. Add to Caddyfile:
|
||||
```
|
||||
authentik.example.com {
|
||||
reverse_proxy authentik-server-1:9000
|
||||
}
|
||||
```
|
||||
|
||||
## Caddyfile Reference
|
||||
|
||||
The Caddyfile supports environment variable substitution:
|
||||
|
||||
```
|
||||
{$FRONTEND_DOMAIN:app.example.com} {
|
||||
reverse_proxy web:3000
|
||||
}
|
||||
|
||||
{$API_DOMAIN:api.example.com} {
|
||||
reverse_proxy server:1250
|
||||
}
|
||||
```
|
||||
|
||||
Set `FRONTEND_DOMAIN` and `API_DOMAIN` environment variables, or edit the file directly.
|
||||
|
||||
### Reload Caddy after changes
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml exec caddy caddy reload --config /etc/caddy/Caddyfile
|
||||
```
|
||||
|
||||
@@ -1,7 +1,171 @@
|
||||
---
|
||||
title: modal setup
|
||||
sidebar_position: 4
|
||||
title: Modal.com Setup
|
||||
---
|
||||
|
||||
# modal setup
|
||||
# Modal.com Setup
|
||||
|
||||
Documentation coming soon. See [TODO.md](/docs/TODO) for required information.
|
||||
This page covers Modal.com GPU setup in detail. For the complete deployment guide, see [Deployment Guide](./overview).
|
||||
|
||||
Reflector uses [Modal.com](https://modal.com) for GPU-accelerated audio processing. This guide walks you through deploying the required GPU functions.
|
||||
|
||||
## What is Modal.com?
|
||||
|
||||
Modal is a serverless GPU platform. You deploy Python code that runs on their GPUs, and pay only for actual compute time. Reflector uses Modal for:
|
||||
|
||||
- **Transcription**: Whisper model for speech-to-text
|
||||
- **Diarization**: Pyannote model for speaker identification
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Modal.com account** - Sign up at https://modal.com (free tier available)
|
||||
2. **HuggingFace account** - Required for Pyannote diarization models:
|
||||
- Create account at https://huggingface.co
|
||||
- Accept **both** Pyannote licenses:
|
||||
- https://huggingface.co/pyannote/speaker-diarization-3.1
|
||||
- https://huggingface.co/pyannote/segmentation-3.0
|
||||
- Generate access token at https://huggingface.co/settings/tokens
|
||||
|
||||
## Deployment
|
||||
|
||||
**Location: YOUR LOCAL COMPUTER (laptop/desktop)**
|
||||
|
||||
Modal CLI requires browser authentication, so this must run on a machine with a browser - not on a headless server.
|
||||
|
||||
### Install Modal CLI
|
||||
|
||||
```bash
|
||||
pip install modal
|
||||
```
|
||||
|
||||
### Authenticate with Modal
|
||||
|
||||
```bash
|
||||
modal setup
|
||||
```
|
||||
|
||||
This opens your browser for authentication. Complete the login flow.
|
||||
|
||||
### Clone Repository and Deploy
|
||||
|
||||
```bash
|
||||
git clone https://github.com/monadical-sas/reflector.git
|
||||
cd reflector/gpu/modal_deployments
|
||||
./deploy-all.sh --hf-token YOUR_HUGGINGFACE_TOKEN
|
||||
```
|
||||
|
||||
Or run interactively (script will prompt for token):
|
||||
```bash
|
||||
./deploy-all.sh
|
||||
```
|
||||
|
||||
### What the Script Does
|
||||
|
||||
1. **Prompts for HuggingFace token** - Needed to download the Pyannote diarization model
|
||||
2. **Generates API key** - Creates a secure random key for authenticating requests to GPU functions
|
||||
3. **Creates Modal secrets**:
|
||||
- `hf_token` - Your HuggingFace token
|
||||
- `reflector-gpu` - The generated API key
|
||||
4. **Deploys GPU functions** - Transcriber (Whisper) and Diarizer (Pyannote)
|
||||
5. **Outputs configuration** - Prints URLs and API key to console
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
==========================================
|
||||
Reflector GPU Functions Deployment
|
||||
==========================================
|
||||
|
||||
Generating API key for GPU services...
|
||||
Creating Modal secrets...
|
||||
-> Creating secret: hf_token
|
||||
-> Creating secret: reflector-gpu
|
||||
|
||||
Deploying transcriber (Whisper)...
|
||||
-> https://yourname--reflector-transcriber-web.modal.run
|
||||
|
||||
Deploying diarizer (Pyannote)...
|
||||
-> https://yourname--reflector-diarizer-web.modal.run
|
||||
|
||||
==========================================
|
||||
Deployment complete!
|
||||
==========================================
|
||||
|
||||
Copy these values to your server's server/.env file:
|
||||
|
||||
# --- Modal GPU Configuration ---
|
||||
TRANSCRIPT_BACKEND=modal
|
||||
TRANSCRIPT_URL=https://yourname--reflector-transcriber-web.modal.run
|
||||
TRANSCRIPT_MODAL_API_KEY=abc123...
|
||||
|
||||
DIARIZATION_BACKEND=modal
|
||||
DIARIZATION_URL=https://yourname--reflector-diarizer-web.modal.run
|
||||
DIARIZATION_MODAL_API_KEY=abc123...
|
||||
# --- End Modal Configuration ---
|
||||
```
|
||||
|
||||
Copy the output and paste it into your `server/.env` file on your server.
|
||||
|
||||
## Costs
|
||||
|
||||
Modal charges based on GPU compute time:
|
||||
- Functions scale to zero when not in use (no cost when idle)
|
||||
- You only pay for actual processing time
|
||||
- Free tier includes $30/month of credits
|
||||
|
||||
Typical costs for audio processing:
|
||||
- Transcription: ~$0.01-0.05 per minute of audio
|
||||
- Diarization: ~$0.02-0.10 per minute of audio
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Modal CLI not installed"
|
||||
```bash
|
||||
pip install modal
|
||||
```
|
||||
|
||||
### "Not authenticated with Modal"
|
||||
```bash
|
||||
modal setup
|
||||
# Complete browser authentication
|
||||
```
|
||||
|
||||
### "Failed to create secret hf_token"
|
||||
- Verify your HuggingFace token is valid
|
||||
- Ensure you've accepted the Pyannote license
|
||||
- Token needs `read` permission
|
||||
|
||||
### Deployment fails
|
||||
Check the Modal dashboard for detailed error logs:
|
||||
- Visit https://modal.com/apps
|
||||
- Click on the failed function
|
||||
- View build and runtime logs
|
||||
|
||||
### Re-running deployment
|
||||
The script is safe to re-run. It will:
|
||||
- Update existing secrets if they exist
|
||||
- Redeploy functions with latest code
|
||||
- Output new configuration (API key stays the same if secret exists)
|
||||
|
||||
## Manual Deployment (Advanced)
|
||||
|
||||
If you prefer to deploy functions individually:
|
||||
|
||||
```bash
|
||||
cd gpu/modal_deployments
|
||||
|
||||
# Create secrets manually
|
||||
modal secret create hf_token HF_TOKEN=your-hf-token
|
||||
modal secret create reflector-gpu REFLECTOR_GPU_APIKEY=$(openssl rand -hex 32)
|
||||
|
||||
# Deploy each function
|
||||
modal deploy reflector_transcriber.py
|
||||
modal deploy reflector_diarizer.py
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
View your deployed functions and their usage:
|
||||
- **Modal Dashboard**: https://modal.com/apps
|
||||
- **Function logs**: Click on any function to view logs
|
||||
- **Usage**: View compute time and costs in the dashboard
|
||||
|
||||
@@ -1,162 +1,325 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
title: Installation Overview
|
||||
title: Deployment Guide
|
||||
---
|
||||
|
||||
# Installation Overview
|
||||
# Deployment Guide
|
||||
|
||||
Reflector is designed for self-hosted deployment, giving you complete control over your infrastructure and data.
|
||||
This guide walks you through deploying Reflector from scratch. Follow these steps in order.
|
||||
|
||||
## Deployment Options
|
||||
## What You'll Set Up
|
||||
|
||||
### Docker Deployment (Recommended)
|
||||
```
|
||||
User --> Caddy (auto-SSL) --> Frontend (Next.js)
|
||||
--> Backend (FastAPI) --> PostgreSQL
|
||||
--> Redis
|
||||
--> Celery Workers --> Modal.com GPU
|
||||
```
|
||||
|
||||
The easiest way to deploy Reflector:
|
||||
- Pre-configured containers
|
||||
- Automated dependency management
|
||||
- Consistent environment
|
||||
- Easy updates
|
||||
## Prerequisites
|
||||
|
||||
### Manual Installation
|
||||
Before starting, you need:
|
||||
|
||||
For custom deployments:
|
||||
- Greater control over configuration
|
||||
- Integration with existing infrastructure
|
||||
- Custom optimization options
|
||||
- Development environments
|
||||
- [ ] **Production server** - Ubuntu 22.04+, 4+ cores, 8GB+ RAM, public IP
|
||||
- [ ] **Two domain names** - e.g., `app.example.com` (frontend) and `api.example.com` (backend)
|
||||
- [ ] **Modal.com account** - Free tier at https://modal.com
|
||||
- [ ] **HuggingFace account** - Free at https://huggingface.co
|
||||
|
||||
## Requirements
|
||||
### Optional (for live meeting rooms)
|
||||
|
||||
### System Requirements
|
||||
- [ ] **Daily.co account** - Free tier at https://dashboard.daily.co
|
||||
- [ ] **AWS S3 bucket** - For Daily.co recording storage
|
||||
|
||||
**Minimum Requirements:**
|
||||
- CPU: 4 cores
|
||||
- RAM: 8 GB
|
||||
- Storage: 50 GB
|
||||
- OS: Ubuntu 20.04+ or similar Linux
|
||||
---
|
||||
|
||||
**Recommended Requirements:**
|
||||
- CPU: 8+ cores
|
||||
- RAM: 16 GB
|
||||
- Storage: 100 GB SSD
|
||||
- GPU: NVIDIA GPU with 8GB+ VRAM (for local processing)
|
||||
## Step 1: Configure DNS
|
||||
|
||||
### Network Requirements
|
||||
**Location: Your domain registrar / DNS provider**
|
||||
|
||||
- Public IP address (for WebRTC)
|
||||
- Ports: 80, 443, 8000, 3000
|
||||
- Domain name (for SSL)
|
||||
- SSL certificate (Let's Encrypt supported)
|
||||
Create A records pointing to your server:
|
||||
```
|
||||
Type: A Name: app Value: <your-server-ip>
|
||||
Type: A Name: api Value: <your-server-ip>
|
||||
```
|
||||
|
||||
## Required Services
|
||||
Verify propagation (wait a few minutes):
|
||||
```bash
|
||||
dig app.example.com +short
|
||||
dig api.example.com +short
|
||||
# Both should return your server IP
|
||||
```
|
||||
|
||||
### Core Services
|
||||
---
|
||||
|
||||
These services are required for basic operation:
|
||||
## Step 2: Deploy Modal GPU Functions
|
||||
|
||||
1. **PostgreSQL** - Primary database
|
||||
2. **Redis** - Message broker and cache
|
||||
3. **Docker** - Container runtime
|
||||
**Location: YOUR LOCAL COMPUTER (laptop/desktop)**
|
||||
|
||||
### GPU Processing
|
||||
Modal requires browser authentication, so this runs locally - not on your server.
|
||||
|
||||
Choose one:
|
||||
- **Modal.com** - Serverless GPU (recommended)
|
||||
- **Local GPU** - Self-hosted GPU processing
|
||||
### Accept HuggingFace Licenses
|
||||
|
||||
### Optional Services
|
||||
Visit both pages and click "Accept":
|
||||
- https://huggingface.co/pyannote/speaker-diarization-3.1
|
||||
- https://huggingface.co/pyannote/segmentation-3.0
|
||||
|
||||
Enhance functionality with:
|
||||
- **AWS S3** - Long-term storage
|
||||
- **Whereby** - Video conferencing rooms
|
||||
- **Authentik** - Enterprise authentication
|
||||
- **Zulip** - Chat integration
|
||||
Then generate a token at https://huggingface.co/settings/tokens
|
||||
|
||||
## Quick Start
|
||||
### Deploy to Modal
|
||||
|
||||
### Using Docker Compose
|
||||
```bash
|
||||
pip install modal
|
||||
modal setup # opens browser for authentication
|
||||
|
||||
git clone https://github.com/monadical-sas/reflector.git
|
||||
cd reflector/gpu/modal_deployments
|
||||
./deploy-all.sh --hf-token YOUR_HUGGINGFACE_TOKEN
|
||||
```
|
||||
|
||||
**Save the output** - copy the configuration block, you'll need it for Step 5.
|
||||
|
||||
See [Modal Setup](./modal-setup) for troubleshooting and details.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Prepare Server
|
||||
|
||||
**Location: YOUR SERVER (via SSH)**
|
||||
|
||||
### Install Docker
|
||||
|
||||
```bash
|
||||
ssh user@your-server-ip
|
||||
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
sudo usermod -aG docker $USER
|
||||
|
||||
# Log out and back in for group changes
|
||||
exit
|
||||
ssh user@your-server-ip
|
||||
|
||||
docker --version # verify
|
||||
```
|
||||
|
||||
### Open Firewall
|
||||
|
||||
```bash
|
||||
sudo ufw allow 80/tcp
|
||||
sudo ufw allow 443/tcp
|
||||
```
|
||||
|
||||
### Clone Repository
|
||||
|
||||
1. Clone the repository:
|
||||
```bash
|
||||
git clone https://github.com/monadical-sas/reflector.git
|
||||
cd reflector
|
||||
```
|
||||
|
||||
2. Navigate to docker directory:
|
||||
---
|
||||
|
||||
## Step 4: Configure Environment
|
||||
|
||||
**Location: YOUR SERVER (via SSH)**
|
||||
|
||||
Reflector has two env files:
|
||||
- `server/.env` - Backend configuration
|
||||
- `www/.env` - Frontend configuration
|
||||
|
||||
### Backend Configuration
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
cp server/env.example server/.env
|
||||
nano server/.env
|
||||
```
|
||||
|
||||
3. Copy and configure environment:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your settings
|
||||
```
|
||||
|
||||
4. Start services:
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
5. Access Reflector:
|
||||
- Frontend: https://your-domain.com
|
||||
- API: https://your-domain.com/api
|
||||
|
||||
## Configuration Overview
|
||||
|
||||
### Essential Configuration
|
||||
|
||||
**Required settings:**
|
||||
```env
|
||||
# Database
|
||||
DATABASE_URL=postgresql://user:pass@localhost/reflector
|
||||
# Database (defaults work with docker-compose.prod.yml)
|
||||
DATABASE_URL=postgresql+asyncpg://reflector:reflector@postgres:5432/reflector
|
||||
|
||||
# Redis
|
||||
REDIS_URL=redis://localhost:6379
|
||||
REDIS_HOST=redis
|
||||
CELERY_BROKER_URL=redis://redis:6379/1
|
||||
CELERY_RESULT_BACKEND=redis://redis:6379/1
|
||||
|
||||
# Modal.com (for GPU processing)
|
||||
TRANSCRIPT_MODAL_API_KEY=your-key
|
||||
DIARIZATION_MODAL_API_KEY=your-key
|
||||
# Your domains
|
||||
BASE_URL=https://api.example.com
|
||||
CORS_ORIGIN=https://app.example.com
|
||||
CORS_ALLOW_CREDENTIALS=true
|
||||
|
||||
# Domain
|
||||
DOMAIN=your-domain.com
|
||||
# Secret key - generate with: openssl rand -hex 32
|
||||
SECRET_KEY=<your-generated-secret>
|
||||
|
||||
# Modal GPU (paste from deploy-all.sh output)
|
||||
TRANSCRIPT_BACKEND=modal
|
||||
TRANSCRIPT_URL=https://yourname--reflector-transcriber-web.modal.run
|
||||
TRANSCRIPT_MODAL_API_KEY=<from-deploy-all.sh-output>
|
||||
|
||||
DIARIZATION_BACKEND=modal
|
||||
DIARIZATION_URL=https://yourname--reflector-diarizer-web.modal.run
|
||||
DIARIZATION_MODAL_API_KEY=<from-deploy-all.sh-output>
|
||||
|
||||
# Auth - disable for initial setup (see Step 9 for authentication)
|
||||
AUTH_BACKEND=none
|
||||
```
|
||||
|
||||
### Security Configuration
|
||||
### Frontend Configuration
|
||||
|
||||
```bash
|
||||
cp www/.env.example www/.env
|
||||
nano www/.env
|
||||
```
|
||||
|
||||
**Required settings:**
|
||||
```env
|
||||
# Your domains
|
||||
SITE_URL=https://app.example.com
|
||||
API_URL=https://api.example.com
|
||||
WEBSOCKET_URL=wss://api.example.com
|
||||
SERVER_API_URL=http://server:1250
|
||||
|
||||
# NextAuth
|
||||
NEXTAUTH_URL=https://app.example.com
|
||||
NEXTAUTH_SECRET=<generate-with-openssl-rand-hex-32>
|
||||
|
||||
# Disable login requirement for initial setup
|
||||
FEATURE_REQUIRE_LOGIN=false
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Configure Caddy
|
||||
|
||||
**Location: YOUR SERVER (via SSH)**
|
||||
|
||||
Edit Caddyfile with your domains:
|
||||
|
||||
```bash
|
||||
nano Caddyfile
|
||||
```
|
||||
|
||||
Replace example.com:
|
||||
```
|
||||
app.example.com {
|
||||
reverse_proxy web:3000
|
||||
}
|
||||
|
||||
api.example.com {
|
||||
reverse_proxy server:1250
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Start Services
|
||||
|
||||
**Location: YOUR SERVER (via SSH)**
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
Wait for containers to start (~30 seconds), then run migrations:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml exec server uv run alembic upgrade head
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Verify Deployment
|
||||
|
||||
### Check services
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml ps
|
||||
# All should show "Up"
|
||||
```
|
||||
|
||||
### Check logs for errors
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml logs server --tail 20
|
||||
docker compose -f docker-compose.prod.yml logs worker --tail 20
|
||||
```
|
||||
|
||||
### Test API
|
||||
```bash
|
||||
curl https://api.example.com/health
|
||||
# Should return: {"status":"healthy"}
|
||||
```
|
||||
|
||||
### Test Frontend
|
||||
- Visit https://app.example.com
|
||||
- You should see the Reflector interface
|
||||
- Try uploading an audio file to test transcription
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Optional - Enable Authentication
|
||||
|
||||
By default, Reflector is open (no login required). To add authentication:
|
||||
|
||||
See [Authentication Setup](./auth-setup) for full Authentik OAuth configuration.
|
||||
|
||||
Quick summary:
|
||||
1. Deploy Authentik on your server
|
||||
2. Create OAuth provider in Authentik
|
||||
3. Update `server/.env`: `AUTH_BACKEND=jwt`
|
||||
4. Update `www/.env`: `FEATURE_REQUIRE_LOGIN=true` + Authentik credentials
|
||||
5. Restart services
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Optional - Enable Live Meeting Rooms
|
||||
|
||||
Live rooms require Daily.co and AWS S3. Add to `server/.env`:
|
||||
|
||||
```env
|
||||
# Authentication
|
||||
REFLECTOR_AUTH_BACKEND=jwt
|
||||
NEXTAUTH_SECRET=generate-strong-secret
|
||||
DEFAULT_VIDEO_PLATFORM=daily
|
||||
DAILY_API_KEY=<from-daily.co-dashboard>
|
||||
DAILY_SUBDOMAIN=<your-daily-subdomain>
|
||||
|
||||
# SSL (handled by Caddy)
|
||||
# Automatic with Let's Encrypt
|
||||
# S3 for recording storage
|
||||
DAILYCO_STORAGE_AWS_BUCKET_NAME=<your-bucket>
|
||||
DAILYCO_STORAGE_AWS_REGION=us-east-1
|
||||
DAILYCO_STORAGE_AWS_ROLE_ARN=<arn:aws:iam::ACCOUNT:role/DailyCo>
|
||||
```
|
||||
|
||||
## Service Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Caddy Reverse Proxy] --> B[Frontend - Next.js]
|
||||
A --> C[Backend - FastAPI]
|
||||
C --> D[PostgreSQL]
|
||||
C --> E[Redis]
|
||||
C --> F[Celery Workers]
|
||||
F --> G[Modal.com GPU]
|
||||
Restart server:
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml restart server worker
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Services won't start
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml logs
|
||||
```
|
||||
|
||||
### CORS errors in browser
|
||||
- Verify `CORS_ORIGIN` in `server/.env` matches your frontend domain exactly (including `https://`)
|
||||
- Restart: `docker compose -f docker-compose.prod.yml restart server`
|
||||
|
||||
### SSL certificate errors
|
||||
- Caddy auto-provisions Let's Encrypt certificates
|
||||
- Ensure ports 80 and 443 are open
|
||||
- Check: `docker compose -f docker-compose.prod.yml logs caddy`
|
||||
|
||||
### Transcription not working
|
||||
- Check Modal dashboard: https://modal.com/apps
|
||||
- Verify URLs in `server/.env` match deployed functions
|
||||
- Check worker logs: `docker compose -f docker-compose.prod.yml logs worker`
|
||||
|
||||
### "Login required" but auth not configured
|
||||
- Set `FEATURE_REQUIRE_LOGIN=false` in `www/.env`
|
||||
- Rebuild frontend: `docker compose -f docker-compose.prod.yml up -d --force-recreate web`
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Review Requirements**: [System Requirements](./requirements)
|
||||
2. **Docker Setup**: [Docker Deployment Guide](./docker-setup)
|
||||
3. **Configure Services**:
|
||||
- [Modal.com Setup](./modal-setup)
|
||||
- [Whereby Setup](./whereby-setup)
|
||||
- [AWS S3 Setup](./aws-setup)
|
||||
4. **Optional Services**:
|
||||
- [Authentik Setup](./authentik-setup)
|
||||
- [Zulip Setup](./zulip-setup)
|
||||
|
||||
## Getting Help
|
||||
|
||||
- [GitHub Issues](https://github.com/monadical-sas/reflector/issues)
|
||||
- [Community Discord](#)
|
||||
- [Modal Setup](./modal-setup) - GPU processing details
|
||||
- [Authentication Setup](./auth-setup) - Authentik OAuth
|
||||
- [System Requirements](./requirements) - Hardware specs
|
||||
|
||||
@@ -5,25 +5,54 @@ title: System Requirements
|
||||
|
||||
# System Requirements
|
||||
|
||||
## Minimum Requirements
|
||||
This page lists hardware and software requirements. For the complete deployment guide, see [Deployment Guide](./overview).
|
||||
|
||||
## Server Requirements
|
||||
|
||||
### Minimum Requirements
|
||||
|
||||
- **CPU**: 4 cores
|
||||
- **RAM**: 8 GB
|
||||
- **Storage**: 50 GB SSD
|
||||
- **OS**: Ubuntu 20.04+ or compatible Linux
|
||||
- **OS**: Ubuntu 22.04+ or compatible Linux
|
||||
- **Network**: Public IP address
|
||||
|
||||
## Recommended Requirements
|
||||
### Recommended Requirements
|
||||
|
||||
- **CPU**: 8+ cores
|
||||
- **RAM**: 16 GB
|
||||
- **Storage**: 100 GB SSD
|
||||
- **GPU**: NVIDIA GPU with 8GB+ VRAM (for local processing)
|
||||
- **Network**: 1 Gbps connection
|
||||
|
||||
## Software Requirements
|
||||
|
||||
- Docker Engine 20.10+
|
||||
- Docker Compose 2.0+
|
||||
- Node.js 18+ (for frontend development)
|
||||
- Python 3.11+ (for backend development)
|
||||
|
||||
## External Services
|
||||
|
||||
### Required
|
||||
|
||||
- **Two domain names** - One for frontend (e.g., `app.example.com`), one for API (e.g., `api.example.com`)
|
||||
- **Modal.com account** - For GPU-accelerated transcription and diarization (free tier available)
|
||||
- **HuggingFace account** - For Pyannote diarization model access
|
||||
|
||||
### Required for Live Meeting Rooms
|
||||
|
||||
- **Daily.co account** - For video conferencing (free tier available at https://dashboard.daily.co)
|
||||
- **AWS S3 bucket + IAM Role** - For Daily.co to store recordings
|
||||
|
||||
### Optional
|
||||
|
||||
- **AWS S3** - For cloud storage of recordings and transcripts
|
||||
- **Authentik** - For SSO/OIDC authentication
|
||||
- **Sentry** - For error tracking
|
||||
|
||||
## Development Requirements
|
||||
|
||||
For local development only (not required for production deployment):
|
||||
|
||||
- Node.js 22+ (for frontend development)
|
||||
- Python 3.12+ (for backend development)
|
||||
- pnpm (for frontend package management)
|
||||
- uv (for Python package management)
|
||||
|
||||
Reference in New Issue
Block a user