MinIO Presigned URL Configuration

This document covers the setup and troubleshooting of MinIO presigned URLs for artifact uploads in BitFlow.

Overview

BitFlow uses MinIO (S3-compatible storage) with presigned URLs to enable secure file uploads from containers and external clients. This allows temporary, authenticated access without exposing MinIO credentials directly to end users.

Architecture

Dual Endpoint Setup

BitFlow uses a dual endpoint architecture for MinIO:

Private Endpoint (minio:9000): Used for internal API operations within the Kubernetes cluster
Public Endpoint (localhost:9000 or external domain): Used for presigned URLs accessible by containers and external clients


# k8s/bitflow-api.yaml
env:
  - name: MINIO_PRIVATE_ENDPOINT
    value: minio:9000
  - name: MINIO_PUBLIC_ENDPOINT
    value: localhost:9000 # or your external MinIO URL
  - name: MINIO_ACCESS_KEY
    value: bitflow
  - name: MINIO_SECRET_KEY
    value: bitflow123

Storage Service Pattern

The API uses a StorageService wrapper that manages dual providers:


type StorageService struct {
    privateProvider StorageProvider // For internal operations
    publicProvider  StorageProvider // For presigned URLs
}

Configuration

1. MinIO Server Setup

The MinIO server runs with root user credentials:


# k8s/minio.yaml
env:
  - name: MINIO_ROOT_USER
    value: bitflow
  - name: MINIO_ROOT_PASSWORD
    value: bitflow123

2. Bucket Configuration

For presigned URLs to work properly, the bucket must have the correct access policy:


# Set bucket to allow uploads (required for presigned URL uploads)
kubectl exec -n bitflow deployment/minio -- mc anonymous set upload local/bitflow-data

Important: The upload policy is the minimum required for presigned URL uploads to work. Other policies may cause “Access Denied” errors:

❌ private - Blocks presigned URL uploads
❌ download - Allows downloads but blocks uploads
✅ upload - Allows uploads and downloads (recommended)
⚠️ public - Full public access (less secure)

3. API Configuration

The API server uses environment variables to configure MinIO access:


env:
  - name: MINIO_ACCESS_KEY
    value: bitflow # Must match MINIO_ROOT_USER
  - name: MINIO_SECRET_KEY
    value: bitflow123 # Must match MINIO_ROOT_PASSWORD
  - name: MINIO_BUCKET
    value: bitflow-data

Presigned URL Generation

Current Implementation

Presigned URLs are generated for a fixed filename to avoid signature mismatches:


// internal/messaging/types.go
artifactPath := fmt.Sprintf("runs/%s/tasks/%s/artifacts/output.tar.gz", runTask.RunID, runTask.ID)
uploadURL, err := storageService.GetPresignedURL(context.Background(), artifactPath, storage.OperationUpload, expiry)

Usage

Containers receive presigned URLs in their task configuration:


{
  "artifact_upload_urls": {
    "base_upload_url": "http://localhost:9000/bitflow-data/runs/.../artifacts/output.tar.gz?X-Amz-Algorithm=...",
    "expires_at": "2025-09-03T12:00:00Z",
    "max_file_size": 1073741824
  }
}

Upload using curl:


curl -X PUT \
  -H "Content-Type: application/octet-stream" \
  -T your-file.txt \
  "PRESIGNED_URL_HERE"

Troubleshooting

Common Issues

1. Access Denied Errors

Symptom:


<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied.</Message>
</Error>

Solutions:

Check bucket policy: Ensure bucket is set to upload policy


kubectl exec -n bitflow deployment/minio -- mc anonymous get local/bitflow-data
kubectl exec -n bitflow deployment/minio -- mc anonymous set upload local/bitflow-data

Verify credentials match: API credentials must match MinIO root user
- API: MINIO_ACCESS_KEY=bitflow, MINIO_SECRET_KEY=bitflow123
- MinIO: MINIO_ROOT_USER=bitflow, MINIO_ROOT_PASSWORD=bitflow123
Check endpoint accessibility: Ensure public endpoint is reachable from where upload is attempted

2. Filename Placeholder Issues

Symptom: URLs containing {filename} or %7Bfilename%7D

Solution: Use fixed filename approach (current implementation uses output.tar.gz)

3. Endpoint URL Errors

Symptom:


Endpoint url cannot have fully qualified paths

Solution: Ensure endpoints don’t include paths, only host:port

✅ Correct: localhost:9000
❌ Incorrect: localhost:9000/minio

4. Network Connectivity

Symptoms: Connection refused, timeouts