cubbi/docs/specs/1_SPECIFICATIONS.md

# MC - Monadical AI Container Tool

## Overview

MC (Monadical Container) is a command-line tool for managing ephemeral
containers that run AI tools and development environments. It works with both
local Docker and a dedicated remote web service that manages containers in a
Docker-in-Docker (DinD) environment.

## Technology Stack

### MC Service
- **Web Framework**: FastAPI for high-performance, async API endpoints
- **Package Management**: uv (Astral) for dependency management
- **Database**: SQLite for development, PostgreSQL for production
- **Container Management**: Docker SDK for Python
- **Authentication**: OAuth 2.0 integration with Authentik

### MC CLI
- **Language**: Python
- **Package Management**: uv for dependency management
- **Distribution**: Standalone binary via PyInstaller or similar
- **Configuration**: YAML for configuration files

## System Architecture

### Components

1. **CLI Tool (`mc`)**: The command-line interface users interact with
2. **MC Service**: A web service that handles remote container execution
3. **Container Drivers**: Predefined container templates for various AI tools

### Architecture Diagram

```
┌─────────────┐           ┌─────────────────────────┐
│             │           │                         │
│  MC CLI     │◄─────────►│  Local Docker Daemon    │
│  (mc)       │           │                         │
│             │           └─────────────────────────┘
└──────┬──────┘
       │
       │ REST API
       │
┌──────▼──────┐           ┌─────────────────────────┐
│             │           │                         │
│ MC Service  │◄─────────►│  Docker-in-Docker       │
│ (Web API)   │           │                         │
│             │           └─────────────────────────┘
└─────────────┘
       │
       ├──────────────┬───────────────┐
       │              │               │
┌──────▼──────┐ ┌─────▼─────┐  ┌──────▼──────┐
│             │ │           │  │             │
│  Fluentd    │ │ Langfuse  │  │ Other       │
│  Logging    │ │ Logging   │  │ Services    │
│             │ │           │  │             │
└─────────────┘ └───────────┘  └─────────────┘
```

## Core Concepts

- **Session**: An active container instance with a specific driver
- **Driver**: A predefined container template with specific AI tools installed
- **Remote**: A configured MC service instance

## User Configuration

MC supports user-specific configuration via a YAML file located at `~/.config/mc/config.yaml`. This provides a way to set default values, store service credentials, and customize behavior without modifying code.

### Configuration File Structure

```yaml
# ~/.config/mc/config.yaml
defaults:
  driver: "goose"  # Default driver to use
  connect: true    # Automatically connect after creating session
  mount_local: true  # Mount local directory by default
  networks: []  # Default networks to connect to (besides mc-network)

services:
  # Service credentials with simplified naming
  # These are mapped to environment variables in containers
  langfuse:
    url: ""  # Will be set by the user
    public_key: "pk-lf-..."
    secret_key: "sk-lf-..."

  openai:
    api_key: "sk-..."

  anthropic:
    api_key: "sk-ant-..."

  openrouter:
    api_key: "sk-or-..."

docker:
  network: "mc-network"  # Default Docker network to use
  socket: "/var/run/docker.sock"  # Docker socket path

remote:
  default: "production"  # Default remote to use
  endpoints:
    production:
      url: "https://mc.monadical.com"
      auth_method: "oauth"
    staging:
      url: "https://mc-staging.monadical.com"
      auth_method: "oauth"

ui:
  colors: true  # Enable/disable colors in terminal output
  verbose: false  # Enable/disable verbose output
  table_format: "grid"  # Table format for session listings
```

### Environment Variable Mapping

The simplified configuration names are mapped to environment variables:

| Config Path | Environment Variable |
|-------------|---------------------|
| `services.langfuse.url` | `LANGFUSE_URL` |
| `services.langfuse.public_key` | `LANGFUSE_INIT_PROJECT_PUBLIC_KEY` |
| `services.langfuse.secret_key` | `LANGFUSE_INIT_PROJECT_SECRET_KEY` |
| `services.openai.api_key` | `OPENAI_API_KEY` |
| `services.anthropic.api_key` | `ANTHROPIC_API_KEY` |
| `services.openrouter.api_key` | `OPENROUTER_API_KEY` |

### Environment Variable Precedence

1. Command-line arguments (`-e KEY=VALUE`) take highest precedence
2. User config file takes second precedence
3. System defaults take lowest precedence

### Security Considerations

- Configuration file permissions are set to 600 (user read/write only)
- Sensitive values can be referenced from environment variables: `${ENV_VAR}`
- API keys and secrets are never logged or displayed in verbose output

### CLI Configuration Commands

```bash
# View entire configuration
mc config list

# Get specific configuration value
mc config get defaults.driver

# Set configuration value (using simplified naming)
mc config set langfuse.url "https://cloud.langfuse.com"
mc config set openai.api_key "sk-..."

# Network configuration
mc config network list                   # List default networks
mc config network add example-network    # Add a network to defaults
mc config network remove example-network # Remove a network from defaults

# Reset configuration to defaults
mc config reset
```

## CLI Tool Commands

### Basic Commands

```bash
# Create a new session locally (shorthand)
mc

# List active sessions on local system
mc session list

# Create a new session locally
mc session create [OPTIONS]

# Create a session with a specific driver
mc session create --driver goose

# Create a session with a specific project repository
mc session create --driver goose --project github.com/hello/private

# Create a session with external networks
mc session create --network teamnet --network othernetwork

# Create a session with a project (shorthand)
mc git@github.com:hello/private

# Close a specific session
mc session close <id>

# Connect to an existing session
mc session connect <id>

# Stop the current session (from inside the container)
mc stop
```

### Remote Management

```bash
# Add a remote MC service
mc remote add <name> <url>

# List configured remote services
mc remote list

# Remove a remote service
mc remote remove <name>

# Authenticate with a remote service
mc -r <remote_name> auth

# Create a session on a remote service
mc -r <remote_name> [session create]

# List sessions on a remote service
mc -r <remote_name> session list
```

### Environment Variables

```bash
# Set environment variables for a session
mc session create -e VAR1=value1 -e VAR2=value2

# Set environment variables for a remote session
mc -r <remote_name> session create -e VAR1=value1
```

### Logging

```bash
# Stream logs from a session
mc session logs <id>

# Stream logs with follow option
mc session logs <id> -f
```

## MC Service Specification

### Overview

The MC Service is a web service that manages ephemeral containers in a Docker-in-Docker environment. It provides a REST API for container lifecycle management, authentication, and real-time log streaming.

### API Endpoints

#### Authentication

```
POST /auth/login        - Initiate Authentik authentication flow
POST /auth/callback     - Handle Authentik OAuth callback
POST /auth/refresh      - Refresh an existing token
POST /auth/logout       - Invalidate current token
```

### Authentik Integration

The MC Service integrates with Authentik at https://authentik.monadical.io using OAuth 2.0:

1. **Application Registration**:
   - MC Service is registered as an OAuth application in Authentik
   - Configured with redirect URI to `/auth/callback`
   - Assigned appropriate scopes for user identification

2. **Authentication Flow**:
   - User initiates authentication via CLI
   - MC CLI opens browser to Authentik authorization URL
   - User logs in through Authentik's interface
   - Authentik redirects to callback URL with authorization code
   - MC Service exchanges code for access and refresh tokens
   - CLI receives and securely stores tokens

3. **Token Management**:
   - Access tokens used for API authorization
   - Refresh tokens used to obtain new access tokens
   - Tokens are encrypted at rest in CLI configuration

#### Sessions

```
GET /sessions - List all sessions
POST /sessions - Create a new session
GET /sessions/{id} - Get session details
DELETE /sessions/{id} - Terminate a session
POST /sessions/{id}/connect - Establish connection to session
GET /sessions/{id}/logs - Stream session logs
```

#### Drivers

```
GET /drivers - List available drivers
GET /drivers/{name} - Get driver details
```

#### Projects

```
GET /projects - List all projects
POST /projects - Add a new project
GET /projects/{id} - Get project details
PUT /projects/{id} - Update project details
DELETE /projects/{id} - Remove a project
```

### Service Configuration

```yaml
# mc-service.yaml
server:
  port: 3000
  host: 0.0.0.0

docker:
  socket: /var/run/docker.sock
  network: mc-network

auth:
  provider: authentik
  url: https://authentik.monadical.io
  clientId: mc-service

logging:
  providers:
    - type: fluentd
      url: http://fluentd.example.com:24224
    - type: langfuse
      url: https://cloud.langfuse.com
      public_key: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY}
      secret_key: ${LANGFUSE_INIT_PROJECT_SECRET_KEY}

drivers:
  - name: goose
    image: monadical/mc-goose:latest
  - name: aider
    image: monadical/mc-aider:latest
  - name: claude-code
    image: monadical/mc-claude-code:latest

projects:
  storage:
    type: encrypted
    key: ${PROJECT_ENCRYPTION_KEY}
  default_ssh_scan:
    - github.com
    - gitlab.com
    - bitbucket.org
```

### Docker-in-Docker Implementation

The MC Service runs in a container with access to the host's Docker socket, allowing it to create and manage sibling containers. This approach provides:

1. Isolation between containers
2. Simple lifecycle management
3. Resource constraints for security

### Connection Handling

For remote connections to containers, the service provides two methods:

1. **WebSocket Terminal**: Browser-based terminal access
2. **SSH Server**: Each container runs an SSH server for CLI access

### Logging Implementation

The MC Service implements log collection and forwarding:

1. Container logs are captured using Docker's logging drivers
2. Logs are forwarded to configured providers (Fluentd, Langfuse)
3. Real-time log streaming is available via WebSockets

## Project Management

### Persistent Project Configuration

MC provides persistent storage for project-specific configurations that need to survive container restarts. This is implemented through a dedicated volume mount and symlink system:

1. **Configuration Storage**:
   - Each project has a dedicated configuration directory on the host at `~/.mc/projects/<project-hash>/config`
   - For projects specified by URL, the hash is derived from the repository URL
   - For local projects, the hash is derived from the absolute path of the local directory
   - This directory is mounted into the container at `/mc-config`

2. **Driver Configuration**:
   - Each driver can specify configuration files/directories that should persist across sessions
   - These are defined in the driver's `mc-driver.yaml` file in the `persistent_configs` section
   - Example for Goose driver:
     ```yaml
     persistent_configs:
       - source: "/app/.goose"         # Path in container
         target: "/mc-config/goose"    # Path in persistent storage
         type: "directory"             # directory or file
         description: "Goose memory and configuration"
     ```

3. **Automatic Symlinking**:
   - During container initialization, the system:
     - Creates all target directories in the persistent storage
     - Creates symlinks from the source paths to the target paths
   - This makes the persistence transparent to the application

4. **Environment Variables**:
   - Container has access to configuration location via environment variables:
     ```
     MC_CONFIG_DIR=/mc-config
     MC_DRIVER_CONFIG_DIR=/mc-config/<driver-name>
     ```

This ensures that important configurations like Goose's memory store, authentication tokens, and other state information persist between container sessions while maintaining isolation between different projects.

### Adding Projects

Users can add projects with associated credentials:

```bash
# Add a project with SSH key
mc project add github.com/hello/private --ssh-key ~/.ssh/id_ed25519

# Add a project with token authentication
mc project add github.com/hello/private --token ghp_123456789

# List all projects
mc project list

# Remove a project
mc project remove github.com/hello/private
```

### Project Configuration

Projects are stored in the MC service and referenced by their repository URL. The configuration includes:

```yaml
# Project configuration
id: github.com/hello/private
url: git@github.com:hello/private.git
type: git
auth:
  type: ssh
  key: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...encrypted key data...
    -----END OPENSSH PRIVATE KEY-----
  public_key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...
```

## Driver Implementation

### Driver Structure

Each driver is a Docker image with a standardized structure:

```
/
├── entrypoint.sh      # Container initialization
├── mc-init.sh         # Standardized initialization script
├── mc-driver.yaml     # Driver metadata and configuration
├── tool/              # AI tool installation
└── ssh/               # SSH server configuration
```

### Standardized Initialization Script

All drivers include a standardized `mc-init.sh` script that handles common initialization tasks:

```bash
#!/bin/bash

# Project initialization
if [ -n "$MC_PROJECT_URL" ]; then
    echo "Initializing project: $MC_PROJECT_URL"

    # Set up SSH key if provided
    if [ -n "$MC_GIT_SSH_KEY" ]; then
        mkdir -p ~/.ssh
        echo "$MC_GIT_SSH_KEY" > ~/.ssh/id_ed25519
        chmod 600 ~/.ssh/id_ed25519
        ssh-keyscan github.com >> ~/.ssh/known_hosts 2>/dev/null
    fi

    # Set up token if provided
    if [ -n "$MC_GIT_TOKEN" ]; then
        git config --global credential.helper store
        echo "https://$MC_GIT_TOKEN:x-oauth-basic@github.com" > ~/.git-credentials
    fi

    # Clone repository
    git clone $MC_PROJECT_URL /app
    cd /app

    # Run project-specific initialization if present
    if [ -f "/app/.mc/init.sh" ]; then
        bash /app/.mc/init.sh
    fi
fi

# Driver-specific initialization continues...
```

### Driver Configuration (mc-driver.yaml)

```yaml
name: goose
description: Goose with MCP servers
version: 1.0.0
maintainer: team@monadical.com

init:
  pre_command: /mc-init.sh
  command: /entrypoint.sh

environment:
  - name: MCP_HOST
    description: MCP server host
    required: true
    default: http://localhost:8000

  - name: GOOSE_ID
    description: Goose instance ID
    required: false

  # Project environment variables
  - name: MC_PROJECT_URL
    description: Project repository URL
    required: false

  - name: MC_PROJECT_TYPE
    description: Project repository type (git, svn, etc.)
    required: false
    default: git

  - name: MC_GIT_SSH_KEY
    description: SSH key for Git authentication
    required: false
    sensitive: true

  - name: MC_GIT_TOKEN
    description: Token for Git authentication
    required: false
    sensitive: true

ports:
  - 8000   # Main application
  - 22     # SSH server

volumes:
  - mountPath: /app
    description: Application directory

persistent_configs:
  - source: "/app/.goose"
    target: "/mc-config/goose"
    type: "directory"
    description: "Goose memory and configuration"
```

### Example Built-in Drivers

1. **goose**: Goose with MCP servers
2. **aider**: Aider coding assistant
3. **claude-code**: Claude Code environment
4. **custom**: Custom Dockerfile support

## Network Management

### Docker Network Integration

MC provides flexible network management for containers:

1. **Default MC Network**:
   - Each container is automatically connected to the MC network (`mc-network` by default)
   - This ensures containers can communicate with each other

2. **External Network Connection**:
   - Containers can be connected to one or more external Docker networks
   - This allows integration with existing infrastructure (e.g., databases, web servers)
   - Networks can be specified at session creation time: `mc session create --network mynetwork`

3. **Default Networks Configuration**:
   - Users can configure default networks in their configuration
   - These networks will be used for all new sessions unless overridden
   - Managed with `mc config network` commands

4. **Network Command Examples**:
   ```bash
   # Use with session creation
   mc session create --network teamnet

   # Use with multiple networks
   mc session create --network teamnet --network dbnet

   # Configure default networks
   mc config network add teamnet
   ```

## Security Considerations

1. **Container Isolation**: Each session runs in an isolated container
2. **Authentication**: Integration with Authentik for secure authentication
3. **Resource Limits**: Configurable CPU, memory, and storage limits
4. **Network Isolation**: Internal Docker network for container-to-container communication with optional external network connections
5. **Encrypted Connections**: TLS for API connections and SSH for terminal access

## Deployment

### MC Service Deployment

```yaml
# docker-compose.yml for MC Service
version: '3.8'

services:
  mc-service:
    image: monadical/mc-service:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config:/app/config
    ports:
      - "3000:3000"
    environment:
      - AUTH_URL=https://authentik.monadical.io
      - LANGFUSE_API_KEY=your_api_key
    networks:
      - mc-network

networks:
  mc-network:
    driver: bridge
```

## Project Repository Integration Workflow

### Adding a Project Repository

1. User adds project repository with authentication:
   ```bash
   mc project add github.com/hello/private --ssh-key ~/.ssh/id_ed25519
   ```

2. MC CLI reads the SSH key, encrypts it, and sends to MC Service

3. MC Service stores the project configuration securely

### Using a Project in a Session

1. User creates a session with a project:
   ```bash
   mc -r monadical git@github.com:hello/private
   ```

2. MC Service:
   - Identifies the project from the URL
   - Retrieves project authentication details
   - Sets up environment variables:
     ```
     MC_PROJECT_URL=git@github.com:hello/private
     MC_PROJECT_TYPE=git
     MC_GIT_SSH_KEY=<contents of the SSH key>
     ```
   - Creates container with these environment variables

3. Container initialization:
   - The standardized `mc-init.sh` script detects the project environment variables
   - Sets up SSH key or token authentication
   - Clones the repository to `/app`
   - Runs any project-specific initialization scripts

4. User can immediately begin working with the repository

## Implementation Roadmap

1. **Phase 1**: Local CLI tool with Docker integration
2. **Phase 2**: MC Service REST API with basic container management
3. **Phase 3**: Authentication and secure connections
4. **Phase 4**: Project management functionality
5. **Phase 5**: Driver implementation (Goose, Aider, Claude Code)
6. **Phase 6**: Logging integration with Fluentd and Langfuse
7. **Phase 7**: CLI remote connectivity improvements
8. **Phase 8**: Additional drivers and extensibility features