Nixpacks + Cloud Registry Integration Guide

This document describes the complete Nixpacks + Cloud Registry integration for BitFlow, enabling automatic container image builds from Git repositories.

🏗️ Architecture Overview

Components

API Server - Receives webhooks, manages builds, queues jobs
Build Worker - Processes build jobs using Nixpacks
RabbitMQ - Reliable job queue with retries and dead letters
Cloud Registry - Secure container image storage (GCR, ECR, ACR, etc.)
Nixpacks - Automatic buildpack detection and building

🚀 Quick Start

1. Prerequisites


# Install dependencies
apt-get install -y docker.io git
 
# Install Nixpacks
curl -sSL https://nixpacks.com/install.sh | bash
 
# Verify installation
nixpacks --version
docker --version

2. Configure Services

RabbitMQ Setup:


docker run -d --name rabbitmq \
  -p 5672:5672 -p 15672:15672 \
  -e RABBITMQ_DEFAULT_USER=bitflow \
  -e RABBITMQ_DEFAULT_PASS=bitflow123 \
  rabbitmq:3-management

Cloud Registry Setup (Google Container Registry example):


# Install Google Cloud SDK
curl https://sdk.cloud.google.com | bash
 
# Authenticate with your Google Cloud account
gcloud auth login
 
# Configure Docker to use GCR
gcloud auth configure-docker
 
# Alternative: Use service account key
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

3. Start Build Worker


# Copy configuration
cp configs/worker.yaml.example configs/worker.yaml
 
# Edit with your settings
vim configs/worker.yaml
 
# Start worker
./build/worker --config configs/worker.yaml

📋 API Endpoints

Build Management

Method	Endpoint	Description
`GET`	`/v1/builds`	List image builds
`POST`	`/v1/builds`	Create new build
`GET`	`/v1/builds/:id`	Get build details
`GET`	`/v1/builds/:id/logs`	View build logs
`POST`	`/v1/builds/:id/cancel`	Cancel running build
`POST`	`/v1/builds/:id/retry`	Retry failed build
`GET`	`/v1/builds/builder-types`	List available builders

Webhook Integration

Method	Endpoint	Description
`POST`	`/v1/webhooks/github/events`	GitHub webhook handler
`POST`	`/v1/webhooks/gitlab/events`	GitLab webhook handler
`POST`	`/v1/webhooks/gitee/events`	Gitee webhook handler

🛠️ CLI Usage

Basic Operations


# List all builds
bitflow builds list
 
# Create manual build
bitflow builds create owner/repo --branch main --tag v1.0.0
 
# Monitor build progress
bitflow builds show abc123-def456
bitflow builds logs abc123-def456 --follow
 
# List available builders
bitflow builds builders

Repository Management


# Connect repository with auto-build
bitflow repos connect owner/repo --platform github
 
# Configure webhook (auto-setup)
bitflow repos init --template python

🔧 Build Configuration

Nixpacks Configuration

Create .nixpacks/nixpacks.toml in your repository:


[variables]
NODE_VERSION = "18"
PYTHON_VERSION = "3.11"
 
[buildpacks]
python = true
node = false
 
[phases.build]
cmd = "pip install -r requirements.txt"
 
[phases.start]
cmd = "python app.py"

Build Request Example


{
  "repository_id": "owner/lungadenosquam",
  "branch": "main",
  "builder_type": "nixpacks",
  "image_tag": "v1.0.0",
  "build_config": {
    "nixpacks_version": "1.0.0",
    "registry_url": "gcr.io/my-project",
    "image_name": "medical/lungadenosquam",
    "build_timeout": 1800,
    "scan_on_push": true,
    "environment": {
      "MODEL_TYPE": "classification",
      "DATASET_PATH": "/data"
    }
  }
}

🔄 Workflow Integration

Using Built Images in Workflows


{
  "name": "lung-cancer-training",
  "tasks": [
    {
      "task_type": "container",
      "configuration": {
        "image": "gcr.io/my-project/medical/lungadenosquam:v1.0.0",
        "resources": {
          "gpu_count": 1,
          "memory": "32Gi"
        },
        "dataset_mounts": [
          {
            "dataset_id": "lung-wsi-dataset",
            "mount_path": "/data",
            "permissions": "ro"
          }
        ]
      }
    }
  ]
}

Auto-Build Triggers

Push to main/master - Automatic build with tag main-auto-abc1234
Manual API call - Custom tag and configuration
Scheduled builds - Future: cron-based rebuilds

🏥 Medical AI Example: LungAdenoSquam

Repository Setup


# Connect the LungAdenoSquam repository
bitflow repos connect bitroc-ai/LungAdenoSquam --platform github
 
# Enable auto-builds (default: enabled)
# Push to main will trigger: gcr.io/my-project/bitroc-ai/lungadenosquam:main-auto-abc1234

Training Workflow


{
  "name": "lung-adenosquam-federated-training",
  "dataset_mounts": [
    { "dataset_id": "lung-wsi-us", "mount_path": "/data/us" },
    { "dataset_id": "lung-wsi-eu", "mount_path": "/data/eu" }
  ],
  "tasks": [
    {
      "task_type": "container",
      "configuration": {
        "image": "gcr.io/my-project/bitroc-ai/lungadenosquam:latest",
        "command": ["python", "train.py", "--federated"],
        "resources": { "gpu_count": 1, "memory": "32Gi" }
      }
    }
  ]
}

Federated Execution

US Datacenter: Uses local lung-wsi-us dataset only
EU Datacenter: Uses local lung-wsi-eu dataset only
Both pull same image: gcr.io/my-project/bitroc-ai/lungadenosquam:latest
Results aggregated: Central orchestrator combines model weights

🔍 Monitoring & Debugging

Build Status Tracking


# Monitor queue status
curl -H "Authorization: Bearer $TOKEN" \
  http://api.bitflow.ai/v1/builds/queue/stats
 
# Check build logs
bitflow builds logs abc123-def456
 
# View worker metrics
docker logs bitflow-worker

Common Issues

Issue	Solution
Nixpacks not found	Install: `curl -sSL https://nixpacks.com/install.sh \| bash`
Registry push fails	Check authentication and permissions
Build timeout	Increase `build_timeout` in BuildConfig
Queue backing up	Scale worker count or add more workers

Log Locations

API Server: Application logs
Build Worker: /var/log/bitflow/worker.log
Build Logs: Database image_builds.build_log field
RabbitMQ: Management UI at :15672

🔒 Security Considerations

Cloud Registry Security

Service Accounts - Use service accounts with minimal required permissions
Project/Namespace Isolation - Separate registries per team/environment
Image Scanning - Enable vulnerability scanning (GCR: Container Analysis API)
Access Control - Use IAM policies for fine-grained access control

Build Security

Isolated Builds - Each build runs in separate directory
Resource Limits - Configure memory/CPU limits per build
Network Isolation - Workers should not access external internet
Secret Management - Use environment variables, not hardcoded secrets

Network Architecture


[Internet] -> [Load Balancer] -> [API Server]
                                      |
[Cloud Registry] <- [Build Workers] <-+-> [RabbitMQ]
                                      |
                               [Database]

📈 Scaling & Performance

Horizontal Scaling


# Scale workers across multiple machines
# Machine 1
./build/worker --config configs/worker-1.yaml
 
# Machine 2
./build/worker --config configs/worker-2.yaml
 
# Configure different work directories to avoid conflicts

Performance Tuning

Concurrent Workers: Start with 2 per CPU core
Build Timeout: 30 minutes for typical projects
RabbitMQ: Use cluster mode for high availability
Cloud Registry: Automatically scales, no configuration needed

Monitoring Metrics

Build queue depth
Average build time
Success/failure rates
Registry storage usage
Worker CPU/memory usage

🔄 Future Enhancements

Multi-stage Builds - Optimize image layers
Cache Management - Build cache sharing between workers
Build Templates - Pre-configured builds for common frameworks
Parallel Builds - Multi-architecture builds (AMD64, ARM64)
Build Artifacts - Store build reports and test results

This integration provides a complete solution for automated container builds in federated learning environments, ensuring data privacy while enabling consistent model training across multiple datacenters. By using cloud registries instead of self-hosted solutions, the system benefits from global CDN distribution, automatic scaling, and enterprise-grade security.