Files
cubbi/tests/test_integration_docker.py
Mathieu Virbel bae951cf7c feat: comprehensive configuration system and environment variable forwarding (#29)
* feat: migrate container configuration from env vars to YAML config files

- Replace environment variable-based configuration with structured YAML config files
- Add Pydantic models for type-safe configuration management in cubbi_init.py
- Update container.py to generate /cubbi/config.yaml and mount into containers
- Simplify goose plugin to extract provider from default model format
- Remove complex environment variable handling in favor of direct config access
- Maintain backward compatibility while enabling cleaner plugin architecture

* feat: optimize goose plugin to only pass required API key for selected model

- Update goose plugin to set only the API key for the provider of the selected model
- Add selective API key configuration for anthropic, openai, google, and openrouter
- Update README.md with comprehensive automated testing documentation
- Add litellm/gpt-oss:120b to test.sh model matrix (now 5 images × 4 models = 20 tests)
- Include single prompt command syntax for each tool in the documentation

* feat: add comprehensive integration tests with pytest parametrization

- Create tests/test_integration.py with parametrized tests for 5 images × 4 models (20 combinations)
- Add pytest configuration to exclude integration tests by default
- Add integration marker for selective test running
- Include help command tests and image availability tests
- Document test usage in tests/README_integration.md

Integration tests cover:
- goose, aider, claudecode, opencode, crush images
- anthropic/claude-sonnet-4-20250514, openai/gpt-4o, openrouter/openai/gpt-4o, litellm/gpt-oss:120b models
- Proper command syntax for each tool
- Success validation with exit codes and completion markers

Usage:
- pytest (regular tests only)
- pytest -m integration (integration tests only)
- pytest -m integration -k "goose" (specific image)

* feat: update OpenCode plugin with perfect multi-provider configuration

- Add global STANDARD_PROVIDERS constant for maintainability
- Support custom providers (with baseURL) vs standard providers
- Custom providers: include npm package, name, baseURL, apiKey, models
- Standard providers: include only apiKey and empty models
- Use direct API key values from cubbi config instead of env vars
- Only add default model to the provider that matches the default model
- Use @ai-sdk/openai-compatible for OpenAI-compatible providers
- Preserve model names without transformation
- All providers get required empty models{} section per OpenCode spec

This ensures OpenCode can properly recognize and use both native
providers (anthropic, openai, google, openrouter) and custom
providers (litellm, etc.) with correct configuration format.

* refactor: model is now a combination of provider/model

* feat: add separate integration test for Claude Code without model config

Claude Code is Anthropic-specific and doesn't require model selection like other tools.
Created dedicated test that verifies basic functionality without model preselection.

* feat: update Claude Code and Crush plugins to use new config system

- Claude Code plugin now uses cubbi_config.providers to get Anthropic API key
- Crush plugin updated to use cubbi_config.providers for provider configuration
- Both plugins maintain backwards compatibility with environment variables
- Consistent plugin structure across all cubbi images

* feat: add environments_to_forward support for images

- Add environments_to_forward field to ImageConfig and Image models
- Update container creation logic to forward specified environment variables from host
- Add environments_to_forward to claudecode cubbi_image.yaml to ensure Anthropic API key is always available
- Claude Code now gets required environment variables regardless of model selection
- This ensures Claude Code works properly even when other models are specified

Fixes the issue where Claude Code couldn't access Anthropic API key when using different model configurations.

* refactor: remove unused environment field from cubbi_image.yaml files

The 'environment' field was loaded but never processed at runtime.
Only 'environments_to_forward' is actually used to pass environment
variables from host to container.

Cleaned up configuration files by removing:
- 72 lines from aider/cubbi_image.yaml
- 42 lines from claudecode/cubbi_image.yaml
- 28 lines from crush/cubbi_image.yaml
- 16 lines from goose/cubbi_image.yaml
- Empty environment: [] from opencode/cubbi_image.yaml

This makes the configuration files cleaner and only contains
fields that are actually used by the system.

* feat: implement environment variable forwarding for aider

Updates aider to automatically receive all relevant environment variables
from the host, similar to how opencode works.

Changes:
- Added environments_to_forward field to aider/cubbi_image.yaml with
  comprehensive list of API keys, configuration, and proxy variables
- Updated aider_plugin.py to use cubbi_config system for provider/model setup
- Environment variables now forwarded automatically during container creation
- Maintains backward compatibility with legacy environment variables

Environment variables forwarded:
- API Keys: OPENAI_API_KEY, ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, etc.
- Configuration: AIDER_MODEL, GIT_* variables, HTTP_PROXY, etc.
- Timezone: TZ for proper log timestamps

Tested: All aider tests pass, environment variables confirmed forwarded.

* refactor: remove unused volumes and init fields from cubbi_image.yaml files

Both 'volumes' and 'init' fields were loaded but never processed at runtime.
These were incomplete implementations that didn't affect container behavior.

Removed from all 5 images:
- volumes: List with mountPath: /app (incomplete, missing host paths)
- init: pre_command and command fields (unused during container creation)

The cubbi_image.yaml files now only contain fields that are actually used:
- Basic metadata (name, description, version, maintainer, image)
- persistent_configs (working functionality)
- environments_to_forward (working functionality where present)

This makes the configuration files cleaner and eliminates confusion
about what functionality is actually implemented.

* refactor: remove unused ImageInit and VolumeMount models

These models were only referenced in the Image model definition but
never used at runtime since we removed all init: and volumes: fields
from cubbi_image.yaml files.

Removed:
- VolumeMount class (mountPath, description fields)
- ImageInit class (pre_command, command fields)
- init: Optional[ImageInit] field from Image model
- volumes: List[VolumeMount] field from Image model

The Image model now only contains fields that are actually used:
- Basic metadata (name, description, version, maintainer, image)
- environment (loaded but unused - kept for future cleanup)
- persistent_configs (working functionality)
- environments_to_forward (working functionality)

This makes the data model cleaner and eliminates dead code.

* feat: add interactive configuration command

Adds `cubbi configure` command for interactive setup of LLM providers
and models through a user-friendly questionnaire interface.

New features:
- Interactive provider configuration (OpenAI, Anthropic, OpenRouter, etc.)
- API key management with environment variable references
- Model selection with provider/model format validation
- Default settings configuration (image, ports, volumes, etc.)
- Added questionary dependency for interactive prompts

Changes:
- Added cubbi/configure.py with full interactive configuration logic
- Added configure command to cubbi/cli.py
- Updated uv.lock with questionary and prompt-toolkit dependencies

Usage: `cubbi configure`

* refactor: update integration tests for current functionality

Updates integration tests to reflect current cubbi functionality:

test_integration.py:
- Simplified image list (removed crush temporarily)
- Updated model list with current supported models
- Removed outdated help command tests that were timing out
- Simplified claudecode test to basic functionality test
- Updated command templates for current tool versions

test_integration_docker.py:
- Cleaned up container management tests
- Fixed formatting and improved readability
- Updated assertion formatting for better error messages

These changes align the tests with the current state of the codebase
and remove tests that were causing timeouts or failures.

* fix: fix temporary file chmod
2025-08-06 21:27:26 -06:00

373 lines
13 KiB
Python

"""
Integration tests for Docker interactions in Cubbi Container.
These tests require Docker to be running.
"""
import subprocess
import time
import uuid
import docker
# Import the requires_docker decorator from conftest
from conftest import requires_docker
def execute_command_in_container(container_id, command):
"""Execute a command in a Docker container and return the output."""
result = subprocess.run(
["docker", "exec", container_id, "bash", "-c", command],
capture_output=True,
text=True,
)
return result.stdout.strip()
def wait_for_container_init(container_id, timeout=5.0, poll_interval=0.1):
start_time = time.time()
while time.time() - start_time < timeout:
try:
# Check if /cubbi/init.status contains INIT_COMPLETE=true
result = execute_command_in_container(
container_id,
"grep -q 'INIT_COMPLETE=true' /cubbi/init.status 2>/dev/null && echo 'COMPLETE' || echo 'PENDING'",
)
if result == "COMPLETE":
return True
except subprocess.CalledProcessError:
# File might not exist yet or container not ready, continue polling
pass
time.sleep(poll_interval)
# Timeout reached
return False
@requires_docker
def test_integration_session_create_with_volumes(
isolate_cubbi_config, test_file_content
):
"""Test creating a session with a volume mount."""
test_file, test_content = test_file_content
session = None
try:
# Get the isolated container manager
container_manager = isolate_cubbi_config["container_manager"]
# Create a session with a volume mount
session = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-volume-{uuid.uuid4().hex[:8]}",
mount_local=False, # Don't mount current directory
volumes={str(test_file): {"bind": "/test/volume_test.txt", "mode": "ro"}},
)
assert session is not None
assert session.status == "running"
# Wait for container initialization to complete
init_success = wait_for_container_init(session.container_id)
assert init_success, "Container initialization timed out"
# Verify the file exists in the container and has correct content
container_content = execute_command_in_container(
session.container_id, "cat /test/volume_test.txt"
)
assert container_content == test_content
finally:
# Clean up the container (use kill for faster test cleanup)
if session and session.container_id:
container_manager.close_session(session.id, kill=True)
@requires_docker
def test_integration_session_create_with_networks(
isolate_cubbi_config, docker_test_network
):
"""Test creating a session connected to a custom network."""
session = None
try:
# Get the isolated container manager
container_manager = isolate_cubbi_config["container_manager"]
# Create a session with the test network
session = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-network-{uuid.uuid4().hex[:8]}",
mount_local=False, # Don't mount current directory
networks=[docker_test_network],
)
assert session is not None
assert session.status == "running"
# Wait for container initialization to complete
init_success = wait_for_container_init(session.container_id)
assert init_success, "Container initialization timed out"
# Verify the container is connected to the test network
# Use inspect to check network connections
import docker
client = docker.from_env()
container = client.containers.get(session.container_id)
container_networks = container.attrs["NetworkSettings"]["Networks"]
# Container should be connected to both the default cubbi-network and our test network
assert docker_test_network in container_networks
# Verify network interface exists in container
network_interfaces = execute_command_in_container(
session.container_id, "ip link show | grep -v 'lo' | wc -l"
)
# Should have at least 2 interfaces (eth0 for cubbi-network, eth1 for test network)
assert int(network_interfaces) >= 2
finally:
# Clean up the container (use kill for faster test cleanup)
if session and session.container_id:
container_manager.close_session(session.id, kill=True)
@requires_docker
def test_integration_session_create_with_ports(isolate_cubbi_config):
"""Test creating a session with port forwarding."""
session = None
try:
# Get the isolated container manager
container_manager = isolate_cubbi_config["container_manager"]
# Create a session with port forwarding
session = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-ports-{uuid.uuid4().hex[:8]}",
mount_local=False, # Don't mount current directory
ports=[8080, 9000], # Forward these ports
)
assert session is not None
assert session.status == "running"
# Verify ports are mapped
assert isinstance(session.ports, dict)
assert 8080 in session.ports
assert 9000 in session.ports
# Verify port mappings are valid (host ports should be assigned)
assert isinstance(session.ports[8080], int)
assert isinstance(session.ports[9000], int)
assert session.ports[8080] > 0
assert session.ports[9000] > 0
# Wait for container initialization to complete
init_success = wait_for_container_init(session.container_id)
assert init_success, "Container initialization timed out"
# Verify Docker port mappings using Docker client
import docker
client = docker.from_env()
container = client.containers.get(session.container_id)
container_ports = container.attrs["NetworkSettings"]["Ports"]
# Verify both ports are exposed
assert "8080/tcp" in container_ports
assert "9000/tcp" in container_ports
# Verify host port bindings exist
assert container_ports["8080/tcp"] is not None
assert container_ports["9000/tcp"] is not None
assert len(container_ports["8080/tcp"]) > 0
assert len(container_ports["9000/tcp"]) > 0
# Verify host ports match session.ports
host_port_8080 = int(container_ports["8080/tcp"][0]["HostPort"])
host_port_9000 = int(container_ports["9000/tcp"][0]["HostPort"])
assert session.ports[8080] == host_port_8080
assert session.ports[9000] == host_port_9000
finally:
# Clean up the container (use kill for faster test cleanup)
if session and session.container_id:
container_manager.close_session(session.id, kill=True)
@requires_docker
def test_integration_session_create_no_ports(isolate_cubbi_config):
"""Test creating a session without port forwarding."""
session = None
try:
# Get the isolated container manager
container_manager = isolate_cubbi_config["container_manager"]
# Create a session without ports
session = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-no-ports-{uuid.uuid4().hex[:8]}",
mount_local=False, # Don't mount current directory
ports=[], # No ports
)
assert session is not None
assert session.status == "running"
# Verify no ports are mapped
assert isinstance(session.ports, dict)
assert len(session.ports) == 0
# Wait for container initialization to complete
init_success = wait_for_container_init(session.container_id)
assert init_success, "Container initialization timed out"
# Verify Docker has no port mappings
import docker
client = docker.from_env()
container = client.containers.get(session.container_id)
container_ports = container.attrs["NetworkSettings"]["Ports"]
# Should have no port mappings (empty dict or None values)
for port_spec, bindings in container_ports.items():
assert bindings is None or len(bindings) == 0
finally:
# Clean up the container (use kill for faster test cleanup)
if session and session.container_id:
container_manager.close_session(session.id, kill=True)
@requires_docker
def test_integration_session_create_with_single_port(isolate_cubbi_config):
"""Test creating a session with a single port forward."""
session = None
try:
# Get the isolated container manager
container_manager = isolate_cubbi_config["container_manager"]
# Create a session with single port
session = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-single-port-{uuid.uuid4().hex[:8]}",
mount_local=False, # Don't mount current directory
ports=[3000], # Single port
)
assert session is not None
assert session.status == "running"
# Verify single port is mapped
assert isinstance(session.ports, dict)
assert len(session.ports) == 1
assert 3000 in session.ports
assert isinstance(session.ports[3000], int)
assert session.ports[3000] > 0
# Wait for container initialization to complete
init_success = wait_for_container_init(session.container_id)
assert init_success, "Container initialization timed out"
client = docker.from_env()
container = client.containers.get(session.container_id)
container_ports = container.attrs["NetworkSettings"]["Ports"]
# Should have exactly one port mapping
port_mappings = {
k: v for k, v in container_ports.items() if v is not None and len(v) > 0
}
assert len(port_mappings) == 1
assert "3000/tcp" in port_mappings
finally:
# Clean up the container (use kill for faster test cleanup)
if session and session.container_id:
container_manager.close_session(session.id, kill=True)
@requires_docker
def test_integration_kill_vs_stop_speed(isolate_cubbi_config):
"""Test that kill is faster than stop for container termination."""
import time
# Get the isolated container manager
container_manager = isolate_cubbi_config["container_manager"]
# Create two identical sessions for comparison
session_stop = None
session_kill = None
try:
# Create first session (will be stopped gracefully)
session_stop = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-stop-{uuid.uuid4().hex[:8]}",
mount_local=False,
ports=[],
)
# Create second session (will be killed)
session_kill = container_manager.create_session(
image_name="goose",
session_name=f"cubbi-test-kill-{uuid.uuid4().hex[:8]}",
mount_local=False,
ports=[],
)
assert session_stop is not None
assert session_kill is not None
# Wait for both containers to initialize
init_success_stop = wait_for_container_init(session_stop.container_id)
init_success_kill = wait_for_container_init(session_kill.container_id)
assert init_success_stop, "Stop test container initialization timed out"
assert init_success_kill, "Kill test container initialization timed out"
# Time graceful stop
start_time = time.time()
container_manager.close_session(session_stop.id, kill=False)
stop_time = time.time() - start_time
session_stop = None # Mark as cleaned up
# Time kill
start_time = time.time()
container_manager.close_session(session_kill.id, kill=True)
kill_time = time.time() - start_time
session_kill = None # Mark as cleaned up
# Kill should be faster than stop (usually by several seconds)
# We use a generous threshold since system performance can vary
assert (
kill_time < stop_time
), f"Kill ({kill_time:.2f}s) should be faster than stop ({stop_time:.2f}s)"
# Verify both methods successfully closed the containers
# (containers should no longer be in the session list)
remaining_sessions = container_manager.list_sessions()
session_ids = [s.id for s in remaining_sessions]
assert session_stop.id if session_stop else "stop-session" not in session_ids
assert session_kill.id if session_kill else "kill-session" not in session_ids
finally:
# Clean up any remaining containers
if session_stop and session_stop.container_id:
try:
container_manager.close_session(session_stop.id, kill=True)
except Exception:
pass
if session_kill and session_kill.container_id:
try:
container_manager.close_session(session_kill.id, kill=True)
except Exception:
pass