* feat: migrate file and live post-processing pipelines from Celery to Hatchet workflow engine
* fix: always force reprocessing
* fix: ci tests with live pipelines
* fix: ci tests with live pipelines
* fix: add auth guards to prevent anonymous access to write endpoints in non-public mode
* test: anon data accessible regardless of guards
* fix: celery test
* Increase max connections
* Classify hard and transient hatchet errors
* Fan out partial success
* Force reprocessing of error transcripts
* Stop retrying on 402 payment required
* Avoid httpx/hatchet timeout race
* Add retry wrapper to get_response for for transient errors
* Add retry backoff
* Return falsy results so get_response won't retry on empty string
* Skip error status in on_workflow_failure when transcript already ended
* Fix precommit issues
* Fail step on first fan-out failure instead of skipping
* fix: local processing instead of http server for cpu
* add fallback token if service worker doesnt work
* chore: rename processors to keep processor pattern up to date and allow other processors to be createed and used with env vars
* feat: enable daily co in selfhosted + only schedule tasks when necessary
* feat: refactor aws storage to be platform agnostic + add local pad tracking with slfhosted support
* fix: switch structured output to tool-call with reflection retry
Replace the two-pass StructuredOutputWorkflow (TreeSummarize → acomplete)
with astructured_predict + reflection retry loop for structured LLM output.
- Enable function-calling mode (is_function_calling_model=True)
- Use astructured_predict with PromptTemplate for first attempt
- On ValidationError/parse failure, retry with reflection feedback
- Add min_length=10 to TopicResponse title/summary fields
- Remove dead StructuredOutputWorkflow class and its event types
- Rewrite tests to match new astructured_predict approach
* fix: include texts parameter in astructured_predict prompt
The switch to astructured_predict dropped the texts parameter entirely,
causing summary prompts (participants, subjects, action items) to be
sent without the transcript content. Combine texts with the prompt
before calling astructured_predict, mirroring what TreeSummarize did.
* fix: reduce TopicResponse min_length from 10 to 8 for title and summary
* ci: try fixing spawning job in github
* ci: fix for new arm64 builder
* feat: local LLM via Ollama + structured output response_format
- Add setup script (scripts/setup-local-llm.sh) for one-command Ollama setup
Mac: native Metal GPU, Linux: containerized via docker-compose profiles
- Add ollama-gpu and ollama-cpu docker-compose profiles for Linux
- Add extra_hosts to server/hatchet-worker-llm for host.docker.internal
- Pass response_format JSON schema in StructuredOutputWorkflow.extract()
enabling grammar-based constrained decoding on Ollama/llama.cpp/vLLM/OpenAI
- Update .env.example with Ollama as default LLM option
- Add Ollama PRD and local dev setup docs
* refactor: move Ollama services to docker-compose.standalone.yml
Ollama profiles (ollama-gpu, ollama-cpu) are only for Linux standalone
deployment. Mac devs never use them. Separate file keeps the main
compose clean and provides a natural home for future standalone services
(MinIO, etc.).
Linux: docker compose -f docker-compose.yml -f docker-compose.standalone.yml --profile ollama-gpu up -d
Mac: docker compose up -d (native Ollama, no standalone file needed)
* fix: correct PRD goal (demo/eval, not dev replacement) and processor naming
* chore: remove completed PRD, rename setup doc, drop response_format tests
- Remove docs/01_ollama.prd.md (implementation complete)
- Rename local-dev-setup.md -> standalone-local-setup.md
- Remove TestResponseFormat class from test_llm_retry.py
* docs: resolve standalone storage step — skip S3 for live-only mode
* docs: add TASKS.md for standalone env defaults + setup script work
* feat: add unified setup-local-dev.sh for standalone deployment
Single script takes fresh clone to working Reflector: Ollama/LLM setup,
env file generation (server/.env + www/.env.local), docker compose up,
health checks. No Hatchet in standalone — live pipeline is pure Celery.
* chore: rename to setup-standalone, remove redundant setup-local-llm.sh
* feat: add custom S3 endpoint support + Garage standalone storage
Add TRANSCRIPT_STORAGE_AWS_ENDPOINT_URL setting to enable S3-compatible
backends (Garage, MinIO). When set, uses path-style addressing and
routes all requests to the custom endpoint. When unset, AWS behavior
is unchanged.
- AwsStorage: accept aws_endpoint_url, pass to all 6 session.client()
calls, configure path-style addressing and base_url
- Fix 4 direct AwsStorage constructions in Hatchet workflows to pass
endpoint_url (would have silently targeted wrong endpoint)
- Standalone: add Garage service to docker-compose.standalone.yml,
setup script initializes layout/bucket/key and writes credentials
- Fix compose_cmd() bug: Mac path was missing standalone yml
- garage.toml template with runtime secret generation via openssl
* fix: standalone setup — garage config, symlink handling, healthcheck
- garage.toml: fix rpc_secret field name (was secret_transmitter),
move to top-level per Garage v1.1.0 spec, remove unused [s3_web]
- setup-standalone.sh: resolve symlinked .env files before writing,
always ensure all standalone-critical vars via env_set,
fix garage key create/info syntax (positional arg, not --name),
avoid overwriting key secret with "(redacted)" on re-run,
use compose_cmd in health check
- docker-compose.standalone.yml: fix garage healthcheck (no curl in
image, use /garage stats instead)
* docs: update standalone md — symlink handling, garage config template
* docs: add troubleshooting section + port conflict check in setup script
Port conflicts from stale next dev / other worktree processes silently
shadow Docker container port mappings, causing env vars to appear ignored.
* fix: invalidate transcript query on STATUS websocket event
Without this, the processing page never redirects after completion
because the redirect logic watches the REST query data, not the
WebSocket status state.
Cherry-picked from feat-dag-progress (faec509a).
* fix: local env setup (#855)
* Ensure rate limit
* Increase nextjs compilation speed
* Fix daily no content handling
* Simplify daily webhook creation
* Fix webhook request validation
* feat: add local pyannote file diarization processor (#858)
* feat: add local pyannote file diarization processor
Enables file diarization without Modal by using pyannote.audio locally.
Downloads model bundle from S3 on first use, caches locally, patches
config to use local paths. Set DIARIZATION_BACKEND=pyannote to enable.
* fix: standalone setup enables pyannote diarization and public mode
Replace DIARIZATION_ENABLED=false with DIARIZATION_BACKEND=pyannote so
file uploads get speaker diarization out of the box. Add PUBLIC_MODE=true
so unauthenticated users can list/browse transcripts.
* fix: touch env files before first compose_cmd in standalone setup
docker-compose.yml references www/.env.local as env_file, but the
setup script only creates it in step 4. compose_cmd calls in step 3
(Garage) fail on a fresh clone when the file doesn't exist yet.
* feat: standalone uses self-hosted GPU service for transcription+diarization
Replace in-process pyannote approach with self-hosted gpu/self_hosted/ service.
Same HTTP API as Modal — just TRANSCRIPT_URL/DIARIZATION_URL point to local container.
- Add gpu/self_hosted/Dockerfile.cpu (GPU Dockerfile minus NVIDIA CUDA)
- Add S3 model bundle fallback in diarizer.py when HF_TOKEN not set
- Add gpu service to docker-compose.standalone.yml with compose env overrides
- Fix /browse empty in PUBLIC_MODE (search+list queries filtered out roomless transcripts)
- Remove audio_diarization_pyannote.py, file_diarization_pyannote.py and tests
- Remove pyannote-audio from server local deps
* fix: allow unauthenticated GPU requests when no API key configured
OAuth2PasswordBearer with auto_error=True rejects requests without
Authorization header before apikey_auth can check if auth is needed.
* fix: rename standalone gpu service to cpu to match Dockerfile.cpu usage
* docs: add programmatic testing section and fix gpu->cpu naming in setup script/docs
- Add "Testing programmatically" section to standalone docs with curl commands
for creating transcript, uploading audio, polling status, checking result
- Fix setup-standalone.sh to reference `cpu` service (was still `gpu` after rename)
- Update all docs references from gpu to cpu service naming
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* Fix websocket disconnect errors
* Fix event loop is closed in Celery workers
* Allow reprocessing idle multitrack transcripts
* feat: add local pyannote file diarization processor
Enables file diarization without Modal by using pyannote.audio locally.
Downloads model bundle from S3 on first use, caches locally, patches
config to use local paths. Set DIARIZATION_BACKEND=pyannote to enable.
* feat: standalone uses self-hosted GPU service for transcription+diarization
Replace in-process pyannote approach with self-hosted gpu/self_hosted/ service.
Same HTTP API as Modal — just TRANSCRIPT_URL/DIARIZATION_URL point to local container.
- Add gpu/self_hosted/Dockerfile.cpu (GPU Dockerfile minus NVIDIA CUDA)
- Add S3 model bundle fallback in diarizer.py when HF_TOKEN not set
- Add gpu service to docker-compose.standalone.yml with compose env overrides
- Fix /browse empty in PUBLIC_MODE (search+list queries filtered out roomless transcripts)
- Remove audio_diarization_pyannote.py, file_diarization_pyannote.py and tests
- Remove pyannote-audio from server local deps
* fix: set source_kind to FILE on audio file upload
The upload endpoint left source_kind as the default LIVE even when
a file was uploaded. Now sets it to FILE when the upload completes.
* Add hatchet env vars
* fix: improve port conflict detection and ollama model check in standalone setup
- Filter OrbStack/Docker Desktop PIDs from port conflict check (false positives on Mac)
- Check all infra ports (5432, 6379, 3900, 3903) not just app ports
- Fix ollama model detection to match on name column only
- Document OrbStack and cross-project port conflicts in troubleshooting
* fix: processing page auto-redirect after file upload completes
Three fixes for the processing page not redirecting when status becomes "ended":
- Add useWebSockets to processing page so it receives STATUS events
- Remove OAuth2PasswordBearer from auth_none — broke WebSocket endpoints (500)
- Reconnect stale Redis in ws_manager when Celery worker reuses dead event loop
* fix: mock Celery broker in idle transcript validation test
test_validation_idle_transcript_with_recording_allowed called
validate_transcript_for_processing without mocking
task_is_scheduled_or_active, which attempts a real Celery
broker connection (AMQP port 5672). Other tests in the same
file already mock this — apply the same pattern here.
* Enable server host mode
* Fix webrtc connection
* Remove turbopack
* fix: standalone GPU service connectivity with host network mode
Server runs with network_mode: host and can't resolve Docker service
names. Publish cpu port as 8100 on host, point server at localhost:8100.
Worker stays on bridge network using cpu:8000. Add dummy
TRANSCRIPT_MODAL_API_KEY since OpenAI SDK requires it even for local
endpoints.
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
Co-authored-by: Sergey Mankovsky <sergey@mankovsky.dev>
* feat: set Daily as default video platform
Daily.co has been battle-tested and is ready to be the default.
Whereby remains available for rooms that explicitly set it.
* feat: enforce Hatchet for all multitrack processing
Remove use_celery option from rooms - multitrack (Daily) recordings
now always use Hatchet workflows. Celery remains for single-track
(Whereby) file processing only.
- Remove use_celery column from room table
- Simplify dispatch logic to always use Hatchet for multitracks
- Update tests to mock Hatchet instead of Celery
* fix: update whereby test to patch Hatchet instead of removed Celery import
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* fix websocket tests
* fix: restore timeout and fix celery test infrastructure
- Re-add timeout=1.0 to ws_manager pubsub loop (prevents CPU spin?)
- Use Redis for Celery tests (memory:// broker doesn't support chords)
- Add timeout param to in-memory subscriber mock
- Remove duplicate celery_includes fixture from rtc_ws tests
* fix: remove redundant inline imports in test files
* fix: update gitleaks ignore for moved s3_key line
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* set hatchet as default for multitracks
* fix: pipeline routing tests for hatchet-default branch
- Create room with use_celery=True to force Celery backend in tests
- Link transcript to room to enable multitrack pipeline routing
- Fixes test failures caused by missing HATCHET_CLIENT_TOKEN in test env
* Update server/reflector/services/transcript_process.py
Co-authored-by: pr-agent-monadical[bot] <198624643+pr-agent-monadical[bot]@users.noreply.github.com>
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
Co-authored-by: pr-agent-monadical[bot] <198624643+pr-agent-monadical[bot]@users.noreply.github.com>
* feat: add transcript format parameter to GET endpoint
Add transcript_format query parameter to /v1/transcripts/{id} endpoint
with support for multiple output formats using discriminated unions.
Formats supported:
- text: Plain speaker dialogue (default)
- text-timestamped: Dialogue with [MM:SS] timestamps
- webvtt-named: WebVTT subtitles with participant names
- json: Structured segments with full metadata
Response models use Pydantic discriminated unions with transcript_format
as discriminator field. POST/PATCH endpoints return GetTranscriptWithParticipants
for minimal responses. GET endpoint returns format-specific models.
* Copy transcript format
* Regenerate types
* Fix transcript formats
* Don't throw inside try
* Remove any type
* Toast share copy errors
* transcript_format exhaustiveness and python idiomatic assert_never
* format_timestamp_mmss clear type definition
* Rename seconds_to_timestamp
* Test transcript format with overlapping speakers
* exact match for vtt multispeaker test
---------
Co-authored-by: Sergey Mankovsky <sergey@monadical.com>
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* Sync authentik users
* Migrate user_id from uid to id
* Fix auth user id
* Fix ci migration test
* Fix meeting token creation
* Move user id migration to a script
* Add user on first login
* Fix migration chain
* Rename uid column to authentik_uid
* Fix broken ws test
* search date filter
* search date filter
* search date filter
* search date filter
* pr comment
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* Update transcript list on reprocess
* Fix transcript create
* Fix multiple sockets issue
* Pass token in sec websocket protocol
* userEvent parse example
* transcript list invalidation non-abstraction
* Emit only relevant events to the user room
* Add ws close code const
* Refactor user websocket endpoint
* Refactor user events provider
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* feat: calendar integration
* feat: add ICS calendar API endpoints for room configuration and sync
* feat: add Celery background tasks for ICS sync
* feat: implement Phase 2 - Multiple active meetings per room with grace period
This commit adds support for multiple concurrent meetings per room, implementing
grace period logic and improved meeting lifecycle management for calendar integration.
## Database Changes
- Remove unique constraint preventing multiple active meetings per room
- Add last_participant_left_at field to track when meeting becomes empty
- Add grace_period_minutes field (default: 15) for configurable grace period
## Meeting Controller Enhancements
- Add get_all_active_for_room() to retrieve all active meetings for a room
- Add get_active_by_calendar_event() to find meetings by calendar event ID
- Maintain backward compatibility with existing get_active() method
## New API Endpoints
- GET /rooms/{room_name}/meetings/active - List all active meetings
- POST /rooms/{room_name}/meetings/{meeting_id}/join - Join specific meeting
## Meeting Lifecycle Improvements
- 15-minute grace period after last participant leaves
- Automatic reactivation when participant rejoins during grace period
- Force close calendar meetings 30 minutes after scheduled end time
- Update process_meetings task to handle multiple active meetings
## Whereby Integration
- Clear grace period when participants join via webhook events
- Track participant count for grace period management
## Testing
- Add comprehensive tests for multiple active meetings
- Test grace period behavior and participant rejoin scenarios
- Test calendar meeting force closure logic
- All 5 new tests passing
This enables proper calendar integration with overlapping meetings while
preventing accidental meeting closures through the grace period mechanism.
* feat: implement frontend for calendar integration (Phase 3 & 4)
- Created MeetingSelection component for choosing between multiple active meetings
- Shows both active meetings and upcoming calendar events (30 min ahead)
- Displays meeting metadata with privacy controls (owner-only details)
- Supports creation of unscheduled meetings alongside calendar meetings
- Added waiting page for users joining before scheduled start time
- Shows countdown timer until meeting begins
- Auto-transitions to meeting when calendar event becomes active
- Handles early joining with proper routing
- Created collapsible info panel showing meeting details
- Displays calendar metadata (title, description, attendees)
- Shows participant count and duration
- Privacy-aware: sensitive info only visible to room owners
- Integrated ICS settings into room configuration dialog
- Test connection functionality with immediate feedback
- Manual sync trigger with detailed results
- Shows last sync time and ETag for monitoring
- Configurable sync intervals (1 min to 1 hour)
- New /room/{roomName} route for meeting selection
- Waiting room at /room/{roomName}/wait?eventId={id}
- Classic room page at /{roomName} with meeting info
- Uses sessionStorage to pass selected meeting between pages
- Added new endpoints for active/upcoming meetings
- Regenerated TypeScript client with latest OpenAPI spec
- Proper error handling and loading states
- Auto-refresh every 30 seconds for live updates
- Color-coded badges for meeting status
- Attendee status indicators (accepted/declined/tentative)
- Responsive design with Chakra UI components
- Clear visual hierarchy between active and upcoming meetings
- Smart truncation for long attendee lists
This completes the frontend implementation for calendar integration,
enabling users to seamlessly join scheduled meetings from their
calendar applications.
* WIP: Migrate calendar integration frontend to React Query
- Migrate all calendar components from useApi to React Query hooks
- Fix Chakra UI v3 compatibility issues (Card, Progress, spacing props, leftIcon)
- Update backend Meeting model to include calendar fields
- Replace imperative API calls with declarative React Query patterns
- Remove old OpenAPI generated files that conflict with new structure
* fix: alembic migrations
* feat: add calendar migration
* feat: update ics, first version working
* feat: implement tabbed interface for room edit dialog
- Add General, Calendar, and Share tabs to organize room settings
- Move ICS settings to dedicated Calendar tab
- Move Zulip configuration to Share tab
- Keep basic room settings and webhooks in General tab
- Remove redundant migration file
- Fix Chakra UI v3 compatibility issues in calendar components
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: infinite loop
* feat: improve ICS calendar sync UX and fix room URL matching
- Replace "Test Connection" button with "Force Sync" button (Edit Room only)
- Show detailed sync results: total events downloaded vs room matches
- Remove emoticons and auto-hide timeout for cleaner UX
- Fix room URL matching to use UI_BASE_URL instead of BASE_URL
- Replace FaSync icon with LuRefreshCw for consistency
- Clear sync results when dialog closes or Force Sync pressed
- Update tests to reflect UI_BASE_URL change and exact URL matching
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: reorganize room edit dialog and fix Force Sync button
- Move WebHook configuration from General to dedicated WebHook tab
- Add WebHook tab after Share tab in room edit dialog
- Fix Force Sync button not appearing by adding missing isEditing prop
- Fix indentation issues in MeetingSelection component
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: complete calendar integration with UI improvements and code cleanup
Calendar Integration Tasks:
- Update upcoming meetings window from 30 to 120 minutes
- Include currently happening events in upcoming meetings API
- Create shared time utility functions (formatDateTime, formatCountdown, formatStartedAgo)
- Improve ongoing meetings UI logic with proper time detection
- Fix backend code organization and remove excessive documentation
UI/UX Improvements:
- Restructure room page layout using MinimalHeader pattern
- Remove borders from header and footer elements
- Change button text from "Leave Meeting" to "Leave Room"
- Remove "Back to Reflector" footer for cleaner design
- Extract WaitPageClient component for better separation
Backend Changes:
- calendar_events.py: Fix import organization and extend timing window
- rooms.py: Update API default from 30 to 120 minutes
- Enhanced test coverage for ongoing meeting scenarios
Frontend Changes:
- MinimalHeader: Add onLeave prop for custom navigation
- MeetingSelection: Complete layout restructure with shared utilities
- timeUtils: New shared utility file for consistent time formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: remove wait page and simplify Join button with 5-minute disable logic
- Remove entire wait page directory and associated files
- Update handleJoinUpcoming to create unscheduled meeting directly
- Simplify Join button to single state:
- Always shows "Join" text
- Blue when meeting can be joined (ongoing or within 5 minutes)
- Gray/disabled when more than 5 minutes away
- Remove confusing "Join Now", "Join Early" text variations
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: improve calendar integration and meeting UI
- Refactor ICS sync tasks to use @asynctask decorator for cleaner async handling
- Extract meeting creation logic into reusable function
- Improve meeting selection UI with distinct current/upcoming sections
- Add early join functionality for upcoming meetings within 5-minute window
- Simplify non-ICS room workflow with direct Whereby embed
- Fix import paths and component organization
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: restore original recording consent functionality
- Remove custom ConsentDialogButton and WherebyEmbed components
- Merge RoomClient logic back into main room page
- Restore original consent UI: blue button with toast modal
- Maintain calendar integration features for ICS-enabled rooms
- Add consent-handler.md documentation of original functionality
- Preserve focus management and accessibility features
* fix: redirect Join Now button to local meeting page
- Change handleJoinDirect to use onMeetingSelect instead of opening external URL
- Join Now button now navigates to /{roomName}/{meetingId} instead of whereby.com
- Maintains proper routing within the application
* feat: remove restrictive message for non-owners in private rooms
- Remove confusing message about room owner permissions
- Cleaner UI for all users regardless of ownership status
- Users will only see available meetings and join options
* feat: improve meeting selection UI for better readability
- Limit page content to max 800px width for better 4K display readability
- Remove LIVE tag badge for cleaner interface
- Remove shadow from main live meeting box
- Remove blue border and hover effects for minimal design
- Change background to neutral gray for less visual noise
* feat: add room by name endpoint for non-authenticated access
- Add GET /rooms/name/{room_name} backend endpoint
- Endpoint supports non-authenticated access for public rooms
- Returns RoomDetails with webhook fields hidden for non-owners
- Update useRoomGetByName hook to use new direct endpoint
- Remove authentication requirement from frontend hook
- Regenerate API client types
Fixes: Non-authenticated users can now access room lobbies
* feat: add friendly message when no meetings are ongoing
- Show centered message with calendar icon when no meetings are active
- Message text: 'No meetings right now' with helpful description
- Contextual text for owners/shared rooms mentioning quick meeting option
- Consistent gray styling matching the rest of the interface
- Only displays when both currentMeetings and upcomingMeetings are empty
* style: center no meetings message and remove background
- Change from Box to Flex with flex=1 for vertical centering
- Remove gray background, border radius, and padding
- Message now appears cleanly centered in available space
- Maintains horizontal and vertical centering
* feat: move Create Meeting button to header
- Remove 'Start a Quick Meeting' box from main content area
- Add showCreateButton and onCreateMeeting props to MinimalHeader
- Create Meeting button now appears in header left of Leave Room
- Only shows for room owners or shared room users
- Update no meetings message to remove reference to quick meeting below
- Cleaner, more accessible UI with actions in the header
* style: change room title and no meetings text to pure black
- Update room title in MinimalHeader from gray.700 to black
- Update 'No meetings right now' text from gray.700 to black
- Improves visual hierarchy and readability
- Consistent with other pages' styling
* style: linting
* fix: remove plan files
* fix: alembic migration with named foreign keys
* feat: add SyncStatus enum and refactor ICS sync to use rooms controller
- Add SyncStatus enum to replace string literals in ICS sync status
- Replace direct SQL queries in worker with rooms_controller.get_ics_enabled()
- Improve type safety and maintainability of ICS sync code
- Enum values: SUCCESS, UNCHANGED, ERROR, SKIPPED maintain backward compatibility
* refactor: remove unnecessary docstring from get_ics_enabled method
The function name is self-explanatory
* fix: import top level
* feat: use Literal type for ICSStatus.status field
- Changed ICSStatus.status from str to Literal['enabled', 'disabled']
- Improves type safety and API documentation
* feat: update TypeScript definitions for ICSStatus Literal type
- OpenAPI generation now properly reflects Literal['enabled', 'disabled'] type
- Improves type safety for frontend consumers of the API
- Applied automatic formatting via pre-commit hooks
* refactor: replace loguru with structlog in ics_sync service
- Replace loguru import with structlog in services/ics_sync.py
- Update logging calls to use structlog's structured format with keyword args
- Maintains consistency with other services using structlog
- Changes: logger.info(f'...') -> logger.info('...', key=value) format
* chore: remove loguru dependency and improve type annotations
- Remove loguru from dependencies in pyproject.toml (replaced with structlog)
- Update meeting controller methods to properly return Optional types
- Update dependency lock file after loguru removal
* fix: resolve pyflakes warnings in ics_sync and meetings modules
Remove unused imports and variables to clean up code quality
* Remove grace period logic and improve meeting deactivation
- Removed grace_period_minutes and last_participant_left_at fields
- Simplified deactivation logic based on actual usage patterns:
* Active sessions: Keep meeting active regardless of scheduled time
* Calendar meetings: Wait until scheduled end if unused, deactivate immediately once used and empty
* On-the-fly meetings: Deactivate immediately when empty
- Created migration to drop unused database columns
- Updated tests to remove grace period test cases
* Update test to match new deactivation logic for calendar meetings
* fix: remove unwanted file
* fix: incompleted changes from EVENT_WINDOW*
* fix: update room ICS API tests to include required webhook fields and correct URL
- Add webhook_url and webhook_secret fields to room creation tests
- Fix room URL matching in ICS sync test to use UI_BASE_URL instead of BASE_URL
- Aligns test with actual API requirements and ICS sync service implementation
* fix: add Redis distributed locking to prevent race conditions in process_meetings
- Implement per-meeting locks using Redis to prevent concurrent processing
- Add lock extension after slow API calls (Whereby) to handle long-running operations
- Use redis-py's built-in lock.extend() with replace_ttl=True for simple TTL refresh
- Track and log skipped meetings when locked by other workers
- Document SSRF analysis showing it's low-risk due to async worker isolation
This prevents multiple workers from processing the same meeting simultaneously,
which could cause state corruption or duplicate deactivations.
* refactor: rename MinimalHeader to MeetingMinimalHeader for clarity
* fix: minor code quality improvements - add emoji constants, fix type safety, cleanup comments
* fix: database migration
* self-pr review
* self-pr review
* self-pr review treeshake
* fix: local fixes
* fix: creation of meeting
* fix: meeting selection create button
* compile fix
* fix: meeting selection responsive
* fix: rework process logic for meeting
* fix: meeting useEffect frontend-only dedupe (#647)
* meeting useEffect frontend-only dedupe
* format
* also get room by name backend fix
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* invalidate meeting list on new meeting
* test fix
* room url copy button for ics
* calendar refresh quick action icon
* remove work.md
* meeting page frontend fixes
* hide number of meeting participants
* Revert "hide number of meeting participants"
This reverts commit 38906c5d1a.
* ui bits
* ui bits
* remove log
* room name typing stricten
* feat: protect atomic operation involving external service with redlock
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Igor Monadical <igor@monadical.com>
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* Self-hosted gpu api
* Refactor self-hosted api
* Rename model api tests
* Use lifespan instead of startup event
* Fix self hosted imports
* Add newlines
* Add response models
* Move gpu dir to the root
* Add project description
* Refactor lifespan
* Update env var names for model api tests
* Preload diarizarion service
* Refactor uploaded file paths
* chore: remove meeting user id and make meeting room id required
* meeting room_id optional
* orphaned meeting room ids DATA migration
* ci fix
* fix meeting_room_id_fkey downgrade
* fix migration rollback
* fix: put index back (meeting room id)
* fix: put index back (meeting room id)
* fix: put index back (meeting room id)
* remove noop migrations
---------
Co-authored-by: Igor Loskutov <igor.loskutoff@gmail.com>
* Documents transcriber api
* Update whisper transcriber api to match parakeet
* Update api transcription spec
* Return 400 for unsupported file type
* Add params to api spec
* Update whisper transcriber implementation to match parakeet
* feat(cleanup): add automatic data retention for public instances
- Add Celery task to clean up anonymous data after configurable retention period
- Delete transcripts, meetings, and orphaned recordings older than retention days
- Only runs when PUBLIC_MODE is enabled to prevent accidental data loss
- Properly removes all associated files (local and S3 storage)
- Add manual cleanup tool for testing and intervention
- Configure retention via PUBLIC_DATA_RETENTION_DAYS setting (default: 7 days)
Fixes#571
* fix: apply pre-commit formatting fixes
* fix: properly delete recording files from storage during cleanup
- Add storage deletion for orphaned recordings in both cleanup task and manual tool
- Delete from storage before removing database records
- Log warnings if storage deletion fails but continue with database cleanup
* Apply suggestion from @pr-agent-monadical[bot]
Co-authored-by: pr-agent-monadical[bot] <198624643+pr-agent-monadical[bot]@users.noreply.github.com>
* Apply suggestion from @pr-agent-monadical[bot]
Co-authored-by: pr-agent-monadical[bot] <198624643+pr-agent-monadical[bot]@users.noreply.github.com>
* refactor: cleanup_old_data for better logging
* fix: linting
* test: fix meeting cleanup test to not require room controller
- Simplify test by directly inserting meetings into database
- Remove dependency on non-existent rooms_controller.create method
- Tests now pass successfully
* fix: linting
* refactor: simplify cleanup tool to use worker implementation
- Remove duplicate cleanup logic from manual tool
- Use the same _cleanup_old_public_data function from worker
- Remove dry-run feature as requested
- Prevent code duplication and ensure consistency
- Update documentation to reflect changes
* refactor: split cleanup worker into smaller functions
- Move all imports to the top of the file
- Extract cleanup logic into separate functions:
- cleanup_old_transcripts()
- cleanup_old_meetings()
- cleanup_orphaned_recordings()
- log_cleanup_results()
- Make code more maintainable and testable
- Add days parameter support to Celery task
- Update manual tool to work with refactored code
* feat: add TypedDict typing for cleanup stats
- Add CleanupStats TypedDict for better type safety
- Update all function signatures to use proper typing
- Add return type annotations to _cleanup_old_public_data
- Improves code maintainability and IDE support
* feat: add CASCADE DELETE to meeting_consent foreign key
- Add ondelete="CASCADE" to meeting_consent.meeting_id foreign key
- Generate and apply migration to update existing constraint
- Remove manual consent deletion from cleanup code
- Add unit test to verify CASCADE DELETE behavior
* style: linting
* fix: alembic migration branchpoint
* fix: correct downgrade constraint name in CASCADE DELETE migration
* fix: regenerate CASCADE DELETE migration with proper constraint names
- Delete problematic migration and regenerate with correct names
- Use explicit constraint name in both upgrade and downgrade
- Ensure migration works bidirectionally
- All tests passing including CASCADE DELETE test
* style: linting
* refactor: simplify cleanup to use transcripts as entry point
- Remove orphaned_recordings cleanup (not part of this PR scope)
- Remove separate old_meetings cleanup
- Transcripts are now the main entry point for cleanup
- Associated meetings and recordings are deleted with their transcript
- Use single database connection for all operations
- Update tests to reflect new approach
* refactor: cleanup and rename functions for clarity
- Rename _cleanup_old_public_data to cleanup_old_public_data (make public)
- Rename celery task to cleanup_old_public_data_task for clarity
- Update docstrings and improve code organization
- Remove unnecessary comments and simplify deletion logic
- Update tests to use new function names
- All tests passing
* style: linting\
* style: typing and review
* fix: add transaction on cleanup_single_transcript
* fix: naming
---------
Co-authored-by: pr-agent-monadical[bot] <198624643+pr-agent-monadical[bot]@users.noreply.github.com>
* feat: use file pipeline for upload and reprocess action
* fix: make file pipeline correctly report status events
* fix: duplication of transcripts_controller
* fix: tests
* test: fix file upload test
* test: fix reprocess
* fix: also patch from main_file_pipeline
(how patch is done is dependent of file import unfortunately)
* feat: improve pipeline threading, and transcriber (parakeet and silero vad)
* refactor: remove whisperx, implement parakeet
* refactor: make audio_chunker more smart and wait for speech, instead of fixed frame
* refactor: make audio merge to always downscale the audio to 16k for transcription
* refactor: make the audio transcript modal accepting batches
* refactor: improve type safety and remove prometheus metrics
- Add DiarizationSegment TypedDict for proper diarization typing
- Replace List/Optional with modern Python list/| None syntax
- Remove all Prometheus metrics from TranscriptDiarizationAssemblerProcessor
- Add comprehensive file processing pipeline with parallel execution
- Update processor imports and type annotations throughout
- Implement optimized file pipeline as default in process.py tool
* refactor: convert FileDiarizationProcessor I/O types to BaseModel
Update FileDiarizationInput and FileDiarizationOutput to inherit from
BaseModel instead of plain classes, following the standard pattern
used by other processors in the codebase.
* test: add tests for file transcript and diarization with pytest-recording
* build: add pytest-recording
* feat: add local pyannote for testing
* fix: replace PyAV AudioResampler with torchaudio for reliable audio processing
- Replace problematic PyAV AudioResampler that was causing ValueError: [Errno 22] Invalid argument
- Use torchaudio.functional.resample for robust sample rate conversion
- Optimize processing: skip conversion for already 16kHz mono audio
- Add direct WAV writing with Python wave module for better performance
- Consolidate duplicate downsample checks for cleaner code
- Maintain list[av.AudioFrame] input interface
- Required for Silero VAD which needs 16kHz mono audio
* fix: replace PyAV AudioResampler with torchaudio solution
- Resolves ValueError: [Errno 22] Invalid argument in AudioMergeProcessor
- Replaces problematic PyAV AudioResampler with torchaudio.functional.resample
- Optimizes processing to skip unnecessary conversions when audio is already 16kHz mono
- Uses direct WAV writing with Python's wave module for better performance
- Fixes test_basic_process to disable diarization (pyannote dependency not installed)
- Updates test expectations to match actual processor behavior
- Removes unused pydub dependency from pyproject.toml
- Adds comprehensive TEST_ANALYSIS.md documenting test suite status
* feat: add parameterized test for both diarization modes
- Adds @pytest.mark.parametrize to test_basic_process with enable_diarization=[False, True]
- Test with diarization=False always passes (tests core AudioMergeProcessor functionality)
- Test with diarization=True gracefully skips when pyannote.audio is not installed
- Provides comprehensive test coverage for both pipeline configurations
* fix: resolve pipeline property naming conflict in AudioDiarizationPyannoteProcessor
- Renames 'pipeline' property to 'diarization_pipeline' to avoid conflict with base Processor.pipeline attribute
- Fixes AttributeError: 'property 'pipeline' object has no setter' when set_pipeline() is called
- Updates property usage in _diarize method to use new name
- Now correctly supports pipeline initialization for diarization processing
* fix: add local for pyannote
* test: add diarization test
* fix: resample on audio merge now working
* fix: correctly restore timestamp
* fix: display exception in a threaded processor if that happen
* Update pyproject.toml
* ci: remove option
* ci: update astral-sh/setup-uv
* test: add monadical url for pytest-recording
* refactor: remove previous version
* build: move faster whisper to local dep
* test: fix missing import
* refactor: improve main_file_pipeline organization and error handling
- Move all imports to the top of the file
- Create unified EmptyPipeline class to replace duplicate mock pipeline code
- Remove timeout and fallback logic - let processors handle their own retries
- Fix error handling to raise any exception from parallel tasks
- Add proper type hints and validation for captured results
* fix: wrong function
* fix: remove task_done
* feat: add configurable file processing timeouts for modal processors
- Add TRANSCRIPT_FILE_TIMEOUT setting (default: 600s) for file transcription
- Add DIARIZATION_FILE_TIMEOUT setting (default: 600s) for file diarization
- Replace hardcoded timeout=600 with configurable settings in modal processors
- Allows customization of timeout values via environment variables
* fix: use logger
* fix: worker process meetings now use file pipeline
* fix: topic not gathered
* refactor: remove prepare(), pipeline now work
* refactor: implement many review from Igor
* test: add test for test_pipeline_main_file
* refactor: remove doc
* doc: add doc
* ci: update build to use native arm64 builder
* fix: merge fixes
* refactor: changes from Igor review + add test (not by default) to test gpu modal part
* ci: update to our own runner linux-amd64
* ci: try using suggested mode=min
* fix: update diarizer for latest modal, and use volume
* fix: modal file extension detection
* fix: put the diarizer as A100
