reflector/TASKS.md

Durable Workflow Migration Tasks

This document defines atomic, isolated work items for migrating the Daily.co multitrack diarization pipeline from Celery to durable workflow orchestration. Supports both Conductor and Hatchet via DURABLE_WORKFLOW_PROVIDER env var.

---

Provider Selection

# .env
DURABLE_WORKFLOW_PROVIDER=none      # Celery only (default)
DURABLE_WORKFLOW_PROVIDER=conductor # Use Conductor
DURABLE_WORKFLOW_PROVIDER=hatchet   # Use Hatchet
DURABLE_WORKFLOW_SHADOW_MODE=true   # Run both provider + Celery (for comparison)

---

Task Index

ID	Title	Status	Conductor	Hatchet
INFRA-001	Add container to docker-compose	Done	✓	✓
INFRA-002	Create Python client wrapper	Done	✓	✓
INFRA-003	Add environment configuration	Done	✓	✓
TASK-001	Create task definitions/workflow	Done	✓ JSON	✓ Python
TASK-002	get_recording worker	Done	✓	✓
TASK-003	get_participants worker	Done	✓	✓
TASK-004	pad_track worker	Done	✓	✓
TASK-005	mixdown_tracks worker	Done	✓	✓
TASK-006	generate_waveform worker	Done	✓	✓
TASK-007	transcribe_track worker	Done	✓	✓
TASK-008	merge_transcripts worker	Done	✓	✓ (in process_tracks)
TASK-009	detect_topics worker	Done	✓	✓
TASK-010	generate_title worker	Done	✓	✓
TASK-011	generate_summary worker	Done	✓	✓
TASK-012	finalize worker	Done	✓	✓
TASK-013	cleanup_consent worker	Done	✓	✓
TASK-014	post_zulip worker	Done	✓	✓
TASK-015	send_webhook worker	Done	✓	✓
EVENT-001	Progress WebSocket events	Done	✓	✓
INTEG-001	Pipeline trigger integration	Done	✓	✓
SHADOW-001	Shadow mode toggle	Done	✓	✓
TEST-001	Integration tests	Pending	-	-
TEST-002	E2E workflow test	Pending	-	-
CUTOVER-001	Production cutover	Pending	-	-
CLEANUP-001	Remove Celery code	Pending	-	-

---

Architecture Differences

Aspect	Conductor	Hatchet
Worker model	Multiprocessing (fork)	Async (single process)
Task communication	REST polling	gRPC streaming
Workflow definition	JSON files	Python decorators
Child workflows	FORK_JOIN_DYNAMIC + JOIN task	aio_run() returns directly
Task definitions	Separate worker files	Embedded in workflow
Debug logging	Limited	Excellent with HATCHET_DEBUG=true

---

File Structure

Conductor

server/reflector/conductor/
├── client.py                    # SDK wrapper
├── progress.py                  # WebSocket progress emission
├── run_workers.py               # Worker startup
├── shadow_compare.py            # Shadow mode comparison
├── tasks/
│   ├── definitions.py           # Task definitions with timeouts
│   └── register.py              # Registration script
├── workers/
│   ├── get_recording.py
│   ├── get_participants.py
│   ├── pad_track.py
│   ├── mixdown_tracks.py
│   ├── generate_waveform.py
│   ├── transcribe_track.py
│   ├── merge_transcripts.py
│   ├── detect_topics.py
│   ├── generate_title.py
│   ├── generate_summary.py
│   ├── finalize.py
│   ├── cleanup_consent.py
│   ├── post_zulip.py
│   ├── send_webhook.py
│   └── generate_dynamic_fork_tasks.py
└── workflows/
    └── register.py

Hatchet

server/reflector/hatchet/
├── client.py                    # SDK wrapper
├── progress.py                  # WebSocket progress emission
├── run_workers.py               # Worker startup
└── workflows/
    ├── diarization_pipeline.py  # Main workflow with all tasks
    └── track_processing.py      # Child workflow (pad + transcribe)

---

Remaining Work

TEST-001: Integration Tests

• Test each worker with mocked external services
• Test error handling and retries
• Test both Conductor and Hatchet paths

TEST-002: E2E Workflow Test

• Complete workflow run with real Daily.co recording
• Verify output matches Celery pipeline
• Performance comparison

CUTOVER-001: Production Cutover

• Deploy with DURABLE_WORKFLOW_PROVIDER=conductor or hatchet
• Monitor for failures
• Compare results with shadow mode if needed

CLEANUP-001: Remove Celery Code

• Remove main_multitrack_pipeline.py
• Remove Celery task triggers
• Update documentation

---

Known Issues

Conductor

• See CONDUCTOR_LLM_OBSERVATIONS.md for debugging notes
• Ghost workers issue (multiple containers polling)
• Multiprocessing + AsyncIO conflicts

Hatchet

• See HATCHET_LLM_OBSERVATIONS.md for debugging notes
• SDK v1.21+ API changes (breaking)
• JWT token Docker networking issues
• Worker appears hung without debug mode

---

Quick Start

Conductor

# Start infrastructure
docker compose up -d conductor conductor-worker

# Register workflow
docker compose exec conductor-worker uv run python -m reflector.conductor.workflows.register

Hatchet

# Start infrastructure
docker compose up -d hatchet hatchet-worker

# Workers auto-register on startup

Trigger Workflow

# Set provider in .env
DURABLE_WORKFLOW_PROVIDER=hatchet  # or conductor

# Process a Daily.co recording via webhook or API
# The pipeline trigger automatically uses the configured provider