LLM Processing Server

Production-ready microservice for asynchronous AI workflow orchestration with comprehensive error handling, external API integration, and intelligent caching.

📋 Table of Contents

What is This?
Key Features
Quick Start
Installation
- Docker Installation (Recommended)
- Local Development Setup
Deployment
- Deploy to Render (Recommended)
- Vercel Deployment
Usage Guide
API Reference
Workflow System
Architecture
Monitoring & Debugging
Advanced Topics
Troubleshooting

What is This?

The LLM Processing Server is a standalone microservice that turns complex AI workflows into simple API calls. Instead of managing LLM integrations, rate limiting, caching, and error handling in your frontend application, offload it all to this dedicated service.

The Problem It Solves

Building AI-powered features in modern applications requires:

Managing multiple LLM providers (OpenAI, Anthropic, etc.)
Integrating external APIs (weather, maps, databases)
Handling long-running async operations
Implementing intelligent caching to reduce costs
Rate limiting to prevent quota exhaustion
Comprehensive error handling and retries
Webhook notifications for job completion
Cost tracking and optimization

The Solution

This service provides a production-ready HTTP API where you:

Submit a workflow → Define multi-step AI operations in JSON
Get a job ID → Track execution via REST API or webhooks
Receive results → Structured output with costs, timing, and metadata

Example: Generate emergency contacts for a location

# Submit workflow
curl -X POST http://your-service.com/api/v1/workflow \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_name": "emergency_contacts",
    "input_data": {
      "city": "San Francisco",
      "lat": 37.7749,
      "lng": -122.4194
    },
    "webhook_url": "https://your-app.com/webhook"
  }'

# Response: Job queued
{
  "job_id": "83134e38-...",
  "status": "queued"
}

# Results delivered to your webhook or via polling

Who Should Use This?

Frontend developers building AI features in Next.js, React, or Vue apps
Backend teams needing a scalable AI processing layer
Startups wanting to avoid building AI infrastructure from scratch
Enterprise teams requiring audit trails, cost tracking, and reliability

Key Features

✅ Core Capabilities

Feature	Description
Multi-Step Workflows	Chain LLM calls, external APIs, and data transformations in a single workflow
LLM Provider Abstraction	Support for Claude, GPT-4, Llama, and 200+ models via OpenRouter
External API Integration	Built-in support for WeatherAPI, Google Places, and custom HTTP endpoints
Async Processing	FastAPI + Celery architecture handles long-running jobs efficiently
Intelligent Caching	Two-level cache (memory + PostgreSQL) with configurable TTLs
Rate Limiting	Per-service token bucket algorithm prevents quota exhaustion
Webhook Notifications	HMAC-signed webhooks with exponential backoff retry logic
Cost Tracking	Token usage and cost calculation for every LLM operation
Comprehensive Error Handling	Structured error context with retry recommendations
Production-Ready	Docker deployment, health checks, monitoring dashboard

📊 Current Status

Phase 9/13 Complete (69%) - Fully functional for workflow execution

Implemented (Phases 1-9):

✅ Core Infrastructure (FastAPI, Celery, Redis, PostgreSQL)
✅ LLM Provider Abstraction (OpenRouter integration)
✅ Workflow Engine (JSON-based workflow execution)
✅ Transformation Library (10+ data transformations)
✅ External API Integration (WeatherAPI, Google Places)
✅ Webhook System (HMAC authentication, retry logic)
✅ Error Handling (Structured errors, retryability detection)
✅ Cache Management (Redis + database caching)
✅ Rate Limiting (Token bucket algorithm)

Planned (Phases 10-13):

🚧 Next.js Integration (TypeScript client, type generation)
🚧 Advanced Monitoring (Prometheus, Grafana)
🚧 Multi-Model Support (Streaming responses, model comparison)
🚧 Advanced Workflows (Conditional branching, parallel execution)

Quick Start

Prerequisites

Docker and Docker Compose (for containerized deployment)
PostgreSQL database (Supabase, Render PostgreSQL, or self-hosted)
API Keys:
- OpenRouter API key (for LLM access)
- WeatherAPI key (optional, for weather workflows)
- Google Places API key (optional)

5-Minute Setup

# 1. Clone the repository
git clone https://github.com/your-org/llm-processing-server.git
cd llm-processing-server

# 2. Copy environment template
cp .env.example .env.local

# 3. Edit .env.local with your credentials
# See .env.example for all available configuration options
nano .env.local  # or use your favorite editor

# 4. Start all services with Docker
docker-compose up --build

# 5. Verify health
curl http://localhost:8000/health

Services Available:

API: http://localhost:8000
API Docs: http://localhost:8000/docs
Flower (Monitoring): http://localhost:5555

Installation

Docker Installation (Recommended)

Docker provides the fastest and most reliable way to run the LLM Processing Server.

Step 1: Install Docker

macOS/Windows:

Download Docker Desktop

Linux:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Step 2: Clone Repository

git clone https://github.com/your-org/llm-processing-server.git
cd llm-processing-server

Step 3: Configure Environment

Create .env.local with your configuration:

# Copy example environment file
cp .env.example .env.local

Required Environment Variables:

#──────────────────────────────────────────────────────────
# DATABASE (Required)
#──────────────────────────────────────────────────────────
DATABASE_URL=postgresql://user:password@host:5432/database

# For Supabase (recommended):
# postgresql://postgres:[PASSWORD]@db.[PROJECT].supabase.co:5432/postgres

# For Render PostgreSQL:
# postgresql://user:password@dpg-xxxxx.oregon-postgres.render.com/database

#──────────────────────────────────────────────────────────
# LLM PROVIDER (Required)
#──────────────────────────────────────────────────────────
OPENROUTER_API_KEY=sk-or-v1-...

# Get your key at: https://openrouter.ai/keys

#──────────────────────────────────────────────────────────
# EXTERNAL APIS (Optional)
#──────────────────────────────────────────────────────────
NEXT_PUBLIC_WEATHERAPI_API_KEY=your-weatherapi-key
NEXT_PUBLIC_GOOGLE_SERVICES_API_KEY=AIza...

#──────────────────────────────────────────────────────────
# REDIS (Auto-configured by Docker)
#──────────────────────────────────────────────────────────
REDIS_URL=redis://redis:6379

#──────────────────────────────────────────────────────────
# WEBHOOKS (Optional)
#──────────────────────────────────────────────────────────
LLM_WEBHOOK_SECRET=your-secret-key-change-in-production

# Generate a secure secret:
# openssl rand -hex 32

#──────────────────────────────────────────────────────────
# EMAIL NOTIFICATIONS (Optional)
#──────────────────────────────────────────────────────────
RESEND_API_KEY=re_...

# Get your key at: https://resend.com/api-keys

#──────────────────────────────────────────────────────────
# ENVIRONMENT
#──────────────────────────────────────────────────────────
ENVIRONMENT=development
DEBUG_MODE=true
LOG_LEVEL=INFO

Step 4: Run Database Migrations

Connect to your PostgreSQL database and run the migrations:

# Using psql (replace with your DATABASE_URL)
psql "postgresql://user:password@host:5432/database" -f migrations/001_create_llm_tables.sql
psql "postgresql://user:password@host:5432/database" -f migrations/002_create_external_api_cache.sql
psql "postgresql://user:password@host:5432/database" -f migrations/003_add_webhook_fields.sql
psql "postgresql://user:password@host:5432/database" -f migrations/004_make_user_id_nullable.sql
psql "postgresql://user:password@host:5432/database" -f migrations/005_restore_constraints.sql
psql "postgresql://user:password@host:5432/database" -f migrations/006_add_username_action.sql

For Supabase:

Go to your Supabase dashboard → SQL Editor
Copy and paste each migration file content
Run them in order

Step 5: Start Services

# Start all services (API, Worker, Redis, Flower)
docker-compose up --build

# Or run in background
docker-compose up -d --build

Docker Compose Services:

Service	Container Name	Port	Description
API	`llm-api`	8000	FastAPI REST API
Worker	`llm-worker`	-	Celery background worker
Redis	`llm-redis`	6379	Message broker & cache
Flower	`llm-flower`	5555	Task monitoring dashboard

Step 6: Verify Installation

# Check health endpoint
curl http://localhost:8000/health

# Expected response:
# {
#   "status": "healthy",
#   "services": {
#     "database": {"status": "healthy"},
#     "redis": {"status": "healthy"},
#     "celery": {"status": "healthy", "workers": 1}
#   }
# }

# View interactive API docs
open http://localhost:8000/docs

# View Flower monitoring dashboard
open http://localhost:5555

Local Development Setup

For development without Docker (requires manual Redis and PostgreSQL setup):

# 1. Create virtual environment
python3.12 -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# 2. Install dependencies
pip install -r requirements.txt

# 3. Start Redis (macOS with Homebrew)
brew services start redis

# 4. Set environment variables
export DATABASE_URL="postgresql://..."
export OPENROUTER_API_KEY="sk-or-v1-..."
export REDIS_URL="redis://localhost:6379"

# 5. Run migrations (as shown in Step 4 above)

# 6. Start services in separate terminals

# Terminal 1: API
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

# Terminal 2: Worker
celery -A app.celery_app worker --loglevel=info --pool=eventlet --concurrency=10 --events

# Terminal 3: Flower (optional)
celery -A app.celery_app flower --port=5555

Deployment

Deploy to Render (Recommended)

Render provides managed services perfect for this architecture: web service, background worker, Redis, and PostgreSQL.

Overview

You'll create 4 services on Render:

PostgreSQL (or use Supabase)
Redis - Message broker
Web Service - FastAPI REST API
Background Worker - Celery task processor
Flower (Optional) - Monitoring dashboard

Estimated Cost: ~$21/month (Starter tier for all services)

Step 1: Create PostgreSQL Database

Option A: Use Supabase (Recommended)

Free tier: 500MB database
Built-in auth and real-time features
Global edge network

Go to supabase.com
Create new project
Copy your database URL from Settings → Database → Connection String (URI)
Format: postgresql://postgres:[PASSWORD]@db.[PROJECT].supabase.co:5432/postgres

Option B: Render PostgreSQL

Go to Render Dashboard
Click New → PostgreSQL
Configure:
- Name: llm-service-db
- Database: llm_service
- User: (auto-generated)
- Region: Oregon (or closest to your users)
- Plan: Starter ($7/month) or Free
Click Create Database
Copy the Internal Database URL (starts with postgresql://)

Step 2: Create Redis Instance

Click New → Redis
Configure:
- Name: llm-service-redis
- Region: Same as your database (Oregon)
- Plan: Starter ($7/month) or Free
- Maxmemory Policy: allkeys-lru (evict least recently used)
Click Create Redis
Copy the Internal Redis URL (e.g., redis://red-xxx:6379)

Step 3: Create Web Service (FastAPI API)

Click New → Web Service
Connect your Git repository or use Deploy an existing image from a registry
Configure:

Basic Settings:
- Name: llm-service-api
- Region: Same as database/Redis
- Branch: main
- Root Directory: Leave empty (or llm_processing_server if it's a subdirectory)
- Runtime: Docker
- Docker Command: uvicorn app.main:app --host 0.0.0.0 --port $PORT
Advanced Settings:
- Plan: Starter ($7/month) - Important: Free tier has 750hr/month limit (not 24/7)
- Health Check Path: /health
- Auto-Deploy: Yes

Environment Variables (click "Advanced" → "Add Environment Variable"):

# Database
DATABASE_URL=<paste-from-step-1>

# Redis (from Step 2)
REDIS_URL=<paste-internal-redis-url>

# LLM Provider
OPENROUTER_API_KEY=sk-or-v1-...

# External APIs (optional)
NEXT_PUBLIC_WEATHERAPI_API_KEY=your-key
NEXT_PUBLIC_GOOGLE_SERVICES_API_KEY=AIza...

# Webhooks
LLM_WEBHOOK_SECRET=your-secret-key

# Environment
ENVIRONMENT=production
DEBUG_MODE=false
LOG_LEVEL=INFO

# Render sets PORT automatically

Click Create Web Service

Note: If you get IPv6 connection issues with Supabase, see IPv6 Troubleshooting

Step 4: Create Background Worker (Celery)

Click New → Background Worker
Use the same repository as the Web Service
Configure:

Basic Settings:
- Name: llm-service-worker
- Region: Same as other services
- Branch: main
- Root Directory: Leave empty
- Runtime: Docker
- Docker Command:
```
celery -A app.celery_app worker --loglevel=info --pool=eventlet --concurrency=10 --events
```
Advanced Settings:
- Plan: Starter ($7/month)
- Auto-Deploy: Yes
Environment Variables: Use the exact same variables as the Web Service (copy-paste from Step 3)
Click Create Background Worker

Step 5: Run Database Migrations

After the API service is deployed:

Go to llm-service-api → Shell (top right)
Run migrations one by one:

# Migration 1: Core tables
psql $DATABASE_URL <<'EOF'
CREATE TABLE IF NOT EXISTS llm_workflow_jobs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  celery_task_id VARCHAR(255) UNIQUE NOT NULL,
  workflow_name VARCHAR(100) NOT NULL,
  user_id UUID,
  status VARCHAR(50) NOT NULL DEFAULT 'pending',
  input_data JSONB,
  result_data JSONB,
  error_message TEXT,
  webhook_url TEXT,
  webhook_secret VARCHAR(255),
  priority INTEGER DEFAULT 0,
  queued_at TIMESTAMPTZ,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  duration_ms INTEGER,
  retry_count INTEGER DEFAULT 0,
  max_retries INTEGER DEFAULT 3,
  debug_mode BOOLEAN DEFAULT false,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_workflow_jobs_status ON llm_workflow_jobs(status);
CREATE INDEX IF NOT EXISTS idx_workflow_jobs_user_id ON llm_workflow_jobs(user_id);
CREATE INDEX IF NOT EXISTS idx_workflow_jobs_workflow_name ON llm_workflow_jobs(workflow_name);
CREATE INDEX IF NOT EXISTS idx_workflow_jobs_created_at ON llm_workflow_jobs(created_at);

-- (Copy all content from migrations/001_create_llm_tables.sql)
EOF

# Repeat for migrations 002-006

For Supabase:

Use the Supabase dashboard → SQL Editor instead
Copy content from each migrations/*.sql file
Run them in order (001 → 006)

Step 6: Create Flower Monitoring (Optional)

Click New → Web Service
Same repository as before
Configure:
- Name: llm-service-flower
- Docker Command: celery -A app.celery_app flower --port=$PORT
- Plan: Free (sufficient for monitoring)
- Same environment variables as other services

Step 7: Verify Deployment

# Replace with your Render URL
API_URL="https://llm-service-api.onrender.com"

# 1. Check health
curl $API_URL/health

# 2. Submit test workflow
curl -X POST $API_URL/api/v1/workflow \
  -H "Content-Type: application/json" \
  -H "X-API-Secret: your-secret" \
  -d '{
    "workflow_name": "emergency_contacts",
    "input_data": {
      "city": "San Francisco",
      "state": "CA",
      "lat": 37.7749,
      "lng": -122.4194,
      "scenarios": ["earthquake"],
      "family_size": 4,
      "duration": "72_hours",
      "user_tier": "BASIC"
    }
  }'

# 3. Check job status (use job_id from response)
curl $API_URL/api/v1/jobs/<job-id> \
  -H "X-API-Secret: your-secret"

Monitor your deployment:

Logs: Go to each service → Logs tab
Metrics: View CPU/Memory usage in Render dashboard
Flower: Open your Flower service URL to see task execution

IPv6 Connectivity Issues

Some platforms (Render, Railway) don't support IPv6 connections to Supabase. If you see errors like:

ENETUNREACH: Network is unreachable

Solution 1: Use Supabase Connection Pooler

# Instead of direct connection:
# postgresql://postgres:pass@db.xxx.supabase.co:5432/postgres

# Use pooler (IPv4):
# postgresql://postgres.xxx:pass@aws-0-us-east-1.pooler.supabase.com:5432/postgres

Solution 2: Use IPv4 Resolver

# In Render Shell, run:
DATABASE_URL="your-supabase-url" node scripts/resolve-db-ipv4.js

# Copy the outputted IPv4 connection string
# Update DATABASE_URL environment variable with it

Note: IP addresses may change. Re-run if connections fail.

Vercel Deployment

⚠️ Important Limitations

Vercel is designed for serverless functions (stateless, short-lived). The LLM Processing Server requires:

❌ Long-running background workers (Celery)
❌ Persistent connections to Redis and PostgreSQL
❌ Execution times > 60 seconds (workflows can take 2+ minutes)
❌ WebSocket support for Flower monitoring

Vercel is NOT recommended for this service.

Alternative Architectures for Vercel

If you must use Vercel, you'll need to split the architecture:

Option 1: Hybrid Deployment

Vercel: Host your Next.js frontend only
Render/Railway: Host this LLM Processing Server
Communication: Frontend calls your Render-hosted API

Option 2: Serverless Adapter (Limited)

Convert API to serverless functions (limited to 60s execution)
Use external Redis (Upstash) and PostgreSQL (Supabase)
Host Celery workers on Render/Railway
Tradeoffs:
- Workflows must complete in <60s
- Cold start latency (1-3s)
- No monitoring dashboard

Option 3: Use Vercel Edge Functions + External Workers

If you want to try (not recommended):

Deploy API as Serverless Functions

Create api/workflow.ts:

// Vercel Serverless Function
import type { VercelRequest, VercelResponse } from '@vercel/node';

export default async function handler(
  req: VercelRequest,
  res: VercelResponse
) {
  if (req.method !== 'POST') {
    return res.status(405).json({ error: 'Method not allowed' });
  }

  // Submit job to external worker (Render/Railway)
  const response = await fetch('https://your-worker-url.onrender.com/api/v1/workflow', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Secret': process.env.API_SECRET!
    },
    body: JSON.stringify(req.body)
  });

  const data = await response.json();
  return res.status(202).json(data);
}

Host Workers Externally

Deploy Celery worker + Redis to Render (as shown above)
Use Supabase for PostgreSQL
Vercel functions proxy requests to worker service

Verdict: Just use Render. It's designed for this architecture and costs the same (~$21/month).

Usage Guide

Basic Workflow Execution

1. Submit a Workflow

POST /api/v1/workflow

# Example: Emergency contacts workflow
curl -X POST http://localhost:8000/api/v1/workflow \
  -H "Content-Type: application/json" \
  -H "X-API-Secret: your-secret-key" \
  -d '{
    "workflow_name": "emergency_contacts",
    "input_data": {
      "city": "San Francisco",
      "state": "CA",
      "lat": 37.7749,
      "lng": -122.4194,
      "scenarios": ["earthquake"],
      "family_size": 4,
      "duration": "72_hours",
      "user_tier": "BASIC",
      "static_contacts": ""
    },
    "webhook_url": "https://your-app.com/api/webhooks/llm",
    "webhook_secret": "optional-per-job-secret"
  }'

Response (202 Accepted):

{
  "job_id": "83134e38-36ec-4e71-977d-2540d54d98c6",
  "status": "queued",
  "workflow_name": "emergency_contacts",
  "created_at": "2025-01-06T12:34:56Z",
  "queued_at": "2025-01-06T12:34:56Z"
}

2. Check Job Status (Polling)

GET /api/v1/jobs/{job_id}

curl -X GET http://localhost:8000/api/v1/jobs/83134e38-36ec-4e71-977d-2540d54d98c6 \
  -H "X-API-Secret: your-secret-key"

Response (200 OK):

{
  "job_id": "83134e38-36ec-4e71-977d-2540d54d98c6",
  "status": "completed",
  "workflow_name": "emergency_contacts",
  "output": {
    "contacts": [
      {
        "category": "Emergency Services",
        "name": "San Francisco Fire Department",
        "phone": "911",
        "address": "Multiple Locations",
        "hours": "24/7",
        "notes": "For immediate life-threatening emergencies"
      }
    ],
    "meeting_locations": [
      {
        "name": "Golden Gate Park - Music Concourse",
        "address": "50 Hagiwara Tea Garden Dr, San Francisco, CA 94118",
        "coordinates": {"lat": 37.7694, "lng": -122.4686},
        "why": "Large open space, easily accessible"
      }
    ]
  },
  "metadata": {
    "duration_ms": 14523,
    "total_tokens": 3842,
    "total_cost_usd": 0.023,
    "steps_completed": 3,
    "cache_hits": 1
  },
  "created_at": "2025-01-06T12:34:56Z",
  "started_at": "2025-01-06T12:34:57Z",
  "completed_at": "2025-01-06T12:35:11Z"
}

3. Webhook Delivery (Recommended)

Instead of polling, receive webhook notifications when jobs complete:

// Your Next.js API route: app/api/webhooks/llm/route.ts
import { NextRequest, NextResponse } from 'next/server';
import crypto from 'crypto';

export async function POST(req: NextRequest) {
  const signature = req.headers.get('x-webhook-signature');
  const event = req.headers.get('x-webhook-event');

  // Read raw body
  const body = await req.text();

  // Verify HMAC signature
  const secret = process.env.LLM_WEBHOOK_SECRET!;
  const expectedSignature = 'sha256=' + crypto
    .createHmac('sha256', secret)
    .update(body)
    .digest('hex');

  if (!crypto.timingSafeEqual(
    Buffer.from(signature || ''),
    Buffer.from(expectedSignature)
  )) {
    return NextResponse.json({ error: 'Invalid signature' }, { status: 401 });
  }

  // Parse payload
  const payload = JSON.parse(body);

  if (event === 'workflow.completed') {
    console.log('Workflow completed:', payload.job_id);
    console.log('Output:', payload.output);
    console.log('Cost:', payload.metadata.total_cost_usd);

    // Update your database
    await db.workflowJobs.update({
      where: { id: payload.job_id },
      data: {
        status: 'completed',
        result: payload.output,
        completedAt: new Date()
      }
    });
  } else if (event === 'workflow.failed') {
    console.error('Workflow failed:', payload.error);

    // Handle error
    await db.workflowJobs.update({
      where: { id: payload.job_id },
      data: {
        status: 'failed',
        error: payload.error.message
      }
    });
  }

  return NextResponse.json({ received: true });
}

Webhook Payload (Success):

{
  "event": "workflow.completed",
  "job_id": "83134e38-36ec-4e71-977d-2540d54d98c6",
  "status": "completed",
  "workflow_name": "emergency_contacts",
  "output": { /* workflow results */ },
  "metadata": {
    "duration_ms": 14523,
    "total_tokens": 3842,
    "total_cost_usd": 0.023
  },
  "created_at": "2025-01-06T12:34:56Z",
  "completed_at": "2025-01-06T12:35:11Z"
}

Webhook Payload (Failure):

{
  "event": "workflow.failed",
  "job_id": "83134e38-36ec-4e71-977d-2540d54d98c6",
  "status": "failed",
  "workflow_name": "emergency_contacts",
  "error": {
    "type": "TimeoutError",
    "category": "EXTERNAL_ERROR",
    "message": "WeatherAPI request timed out",
    "retryable": true,
    "retry_after": 30,
    "step_id": "fetch_weather",
    "details": {
      "service": "weatherapi",
      "operation": "current",
      "suggestions": [
        "Retry the request after 30 seconds",
        "Check WeatherAPI service status"
      ]
    }
  },
  "created_at": "2025-01-06T12:34:56Z",
  "failed_at": "2025-01-06T12:35:26Z"
}

List Jobs

GET /api/v1/jobs?status=completed&limit=10

curl "http://localhost:8000/api/v1/jobs?status=completed&limit=10" \
  -H "X-API-Secret: your-secret-key"

Response:

{
  "jobs": [
    {
      "job_id": "...",
      "status": "completed",
      "workflow_name": "emergency_contacts",
      "created_at": "2025-01-06T12:34:56Z"
    }
  ],
  "total": 42,
  "page": 1,
  "limit": 10
}

Delete Jobs

DELETE /api/v1/jobs/bulk

curl -X DELETE http://localhost:8000/api/v1/jobs/bulk \
  -H "Content-Type: application/json" \
  -H "X-API-Secret: your-secret-key" \
  -d '{
    "job_ids": ["job-id-1", "job-id-2"],
    "status": "completed"
  }'

API Reference

For complete API documentation, see:

Interactive Docs: http://localhost:8000/docs (Swagger UI)
ReDoc: http://localhost:8000/redoc
API Guide: API_GUIDE.md

Authentication

All endpoints (except /health) require the X-API-Secret header:

X-API-Secret: your-secret-key

Set LLM_WEBHOOK_SECRET in your environment variables.

Rate Limits

Tier	Requests/Minute	Concurrent Jobs
Development	10	5
Production	100	20

Configurable via RATE_LIMIT_PER_USER and RATE_LIMIT_GLOBAL environment variables.

Workflow System

What is a Workflow?

A workflow is a JSON file that defines a sequence of steps to execute:

{
  "name": "my_workflow",
  "version": "1.0.0",
  "description": "My custom workflow",
  "timeout_seconds": 120,
  "steps": [
    {
      "id": "step_1",
      "type": "external_api",
      "config": { /* step configuration */ }
    },
    {
      "id": "step_2",
      "type": "llm",
      "config": { /* LLM configuration */ }
    }
  ],
  "outputs": {
    "result": "${steps.step_2.text}"
  }
}

Step Types

Type	Description	Example Use Cases
`llm`	Call language models (Claude, GPT, etc.)	Text generation, analysis, summarization
`external_api`	Call third-party APIs	Weather data, location search, custom APIs
`transform`	Transform data	Parse JSON, format templates, filter data
`conditional`	Branch based on conditions	Route based on results (coming soon)
`parallel`	Execute steps concurrently	Fetch multiple APIs simultaneously (coming soon)

Variable Resolution

Use ${source.path} syntax to reference data:

{
  "variables": {
    "weather": "${steps.fetch_weather.data.current}",
    "location": "${input.city}, ${input.state}",
    "previous_result": "${steps.analyze.text}"
  }
}

Variable Sources:

${input.*} - Workflow input data
${steps.<step_id>.data} - Step output data
${steps.<step_id>.text} - LLM response text
${metadata} - Execution metadata

Creating Custom Workflows

See comprehensive guides:

WORKFLOW_DESIGN.md - Design principles
WORKFLOW_SCHEMA.md - JSON schema reference
Example Workflows - Working examples

Architecture

System Components

┌─────────────────────────────────────────────────────────────┐
│                      Your Application                       │
│              (Next.js, React, Vue, etc.)                    │
└──────────────────────┬──────────────────────────────────────┘
                       │ HTTP POST /api/v1/workflow
                       │ Webhook notifications
                       ▼
┌──────────────────────────────────────────────────────────────┐
│                   FastAPI REST API                           │
│  • Request validation (Pydantic)                            │
│  • Job management & status queries                          │
│  • Health checks & monitoring                               │
└──────────────────────┬──────────────────────────────────────┘
                       │ Celery Tasks
                       ▼
┌──────────────────────────────────────────────────────────────┐
│                  Celery Worker Pool                          │
│  • Async task execution (eventlet)                          │
│  • Workflow orchestration engine                            │
│  • Error handling & retry logic                             │
└────────┬─────────────┬─────────────┬────────────────────────┘
         │             │             │
         ▼             ▼             ▼
  ┌──────────┐  ┌──────────┐  ┌──────────────┐
  │PostgreSQL│  │  Redis   │  │External APIs │
  │          │  │          │  │              │
  │• Jobs    │  │• Queue   │  │• OpenRouter  │
  │• Results │  │• Cache   │  │• WeatherAPI  │
  │• Costs   │  │• Locks   │  │• Google Maps │
  └──────────┘  └──────────┘  └──────────────┘

Technology Stack

Layer	Technology	Purpose
API Framework	FastAPI 0.109.0	REST API with automatic OpenAPI docs
Task Queue	Celery 5.3.6	Async background job processing
Message Broker	Redis 5.0.1	Task queue, caching, rate limiting
Database	PostgreSQL	Job storage, results, audit logs
ASGI Server	Uvicorn 0.27.0	High-performance async server
Worker Pool	Eventlet 0.36.1	Green threads for async operations
Monitoring	Flower 2.0.1	Real-time task monitoring dashboard
Container	Docker	Consistent deployment environment

Request Flow

Client submits workflow → FastAPI validates request
FastAPI creates job → Stores in PostgreSQL with status "queued"
Celery task created → Pushed to Redis queue
Worker picks up task → Executes workflow steps sequentially
Each step executes:
- External API: Check cache → Make request → Cache result
- LLM: Load prompt → Call OpenRouter → Track tokens/cost
- Transform: Apply transformation logic
Workflow completes → Update job status, send webhook
Client receives notification → Webhook or polling

Scaling Strategy

Horizontal Scaling:

Add more Celery workers (increase concurrency)
Use Redis Cluster for distributed caching
PostgreSQL read replicas for query offloading

Vertical Scaling:

Increase worker concurrency (10 → 50 per worker)
Larger Redis instance for cache
PostgreSQL connection pooling

Recommended Production Setup:

3+ Celery workers (concurrency=20 each)
Redis with 2GB+ memory
PostgreSQL with connection pooler (e.g., Supabase Pooler)

Monitoring & Debugging

Flower Dashboard

Access real-time task monitoring: http://localhost:5555

Features:

✅ Active tasks and worker status
✅ Task success/failure rates
✅ Execution time histograms
✅ Worker resource usage (CPU, memory)
✅ Task retry history
✅ Queue inspection

Health Checks

curl http://localhost:8000/health

Checks:

PostgreSQL connectivity
Redis connectivity
Celery worker availability
Service versions

Logs

# Docker logs
docker logs llm-api -f              # API logs
docker logs llm-worker -f           # Worker logs
docker logs llm-redis -f            # Redis logs

# Production (Render)
# View logs in Render dashboard → Service → Logs tab

Database Queries

-- Recent jobs
SELECT job_id, status, workflow_name, created_at, duration_ms
FROM llm_workflow_jobs
ORDER BY created_at DESC
LIMIT 20;

-- Failed jobs
SELECT job_id, workflow_name, error_message, created_at
FROM llm_workflow_jobs
WHERE status = 'failed'
ORDER BY created_at DESC;

-- Cost analysis
SELECT
  workflow_name,
  COUNT(*) as executions,
  AVG(duration_ms) as avg_duration,
  SUM((result_data->'metadata'->>'total_cost_usd')::decimal) as total_cost
FROM llm_workflow_jobs
WHERE status = 'completed'
GROUP BY workflow_name
ORDER BY total_cost DESC;

-- Cache hit rate
SELECT
  service,
  operation,
  COUNT(*) as cache_hits,
  AVG(EXTRACT(EPOCH FROM (expires_at - created_at))) as avg_ttl_seconds
FROM external_api_cache
WHERE expires_at > NOW()
GROUP BY service, operation;

Advanced Topics

Custom Workflows

Create custom workflows by adding JSON files to workflows/definitions/:

{
  "name": "sentiment_analysis",
  "version": "1.0.0",
  "description": "Analyze text sentiment",
  "steps": [
    {
      "id": "analyze",
      "type": "llm",
      "config": {
        "model": "anthropic/claude-3-haiku",
        "prompt_template": "sentiment/analyze.md",
        "variables": {
          "text": "${input.text}"
        },
        "temperature": 0.3,
        "max_tokens": 500
      }
    }
  ],
  "outputs": {
    "sentiment": "${steps.analyze.text}"
  }
}

Create prompt template at workflows/prompts/sentiment/analyze.md:

Analyze the sentiment of the following text. Respond with only one word: positive, negative, or neutral.

Text: {{text}}

Sentiment:

Environment-Specific Configuration

Create multiple environment files:

.env.development     # Local development
.env.staging         # Staging environment
.env.production      # Production environment

Switch environments:

export ENVIRONMENT=production

Cost Optimization

Enable Aggressive Caching:

# In .env
CACHE_DEFAULT_TTL=604800  # 7 days in seconds

Use Cheaper Models:

{
  "model": "anthropic/claude-3-haiku"  // $0.25/1M tokens vs $3/1M for Sonnet
}

Monitor Costs:

SELECT
  DATE(created_at) as date,
  SUM((result_data->'metadata'->>'total_cost_usd')::decimal) as daily_cost
FROM llm_workflow_jobs
WHERE created_at > NOW() - INTERVAL '30 days'
GROUP BY DATE(created_at)
ORDER BY date DESC;

Troubleshooting

Common Issues

1. Redis Connection Refused

Error: Connection refused: redis://redis:6379

Solution:

# Check if Redis is running
docker ps | grep redis

# Restart Redis
docker-compose restart redis

# Check Redis logs
docker logs llm-redis -f

2. Database Connection Pool Exhausted

Error: FATAL: remaining connection slots are reserved

Solution:

Use Supabase connection pooler (Transaction mode)

Reduce SQLAlchemy pool size:

engine = create_engine(
    DATABASE_URL,
    pool_size=5,
    max_overflow=10,
    pool_pre_ping=True
)

3. Event Loop Closed

Error: RuntimeError: Event loop is closed

Solution: Ensure Celery worker uses eventlet pool:

celery -A app.celery_app worker --pool=eventlet --concurrency=10

4. Webhook Signature Validation Fails

Error: Invalid signature on webhook receiver

Solution:

Verify webhook secret matches on both sides
Use raw request body for signature verification (before JSON parsing)
Check for encoding issues (UTF-8)

// Correct signature verification
const signature = req.headers.get('x-webhook-signature');
const rawBody = await req.text();  // Raw body, not parsed JSON

const expectedSig = 'sha256=' + crypto
  .createHmac('sha256', process.env.LLM_WEBHOOK_SECRET!)
  .update(rawBody)
  .digest('hex');

const isValid = crypto.timingSafeEqual(
  Buffer.from(signature || ''),
  Buffer.from(expectedSig)
);

5. IPv6 Connection to Supabase Fails

Error: ENETUNREACH: Network is unreachable

Solution: Use Supabase connection pooler with IPv4:

# Instead of:
postgresql://postgres:pass@db.xxx.supabase.co:5432/postgres

# Use:
postgresql://postgres.xxx:pass@aws-0-us-east-1.pooler.supabase.com:5432/postgres

Debug Mode

Enable detailed logging:

# In .env
DEBUG_MODE=true
LOG_LEVEL=DEBUG

# Restart services
docker-compose restart

Support & Resources

Documentation: Full docs directory
Getting Started: Quick Start, Installation
API Reference: API_GUIDE.md - Complete API documentation
CLI Tools: CLI_GUIDE.md - Workflow development and validation tools
Configuration: .env.example - Environment variable template
Testing Guide: TESTING_GUIDE.md - Test suite documentation
Deployment Guide: DEPLOYMENT_GUIDE.md - Production deployment
Workflow Schema: WORKFLOW_SCHEMA.md - JSON schema reference
GitHub Issues: Create an issue

License

MIT License - see LICENSE for details.

Contributing

Contributions welcome! Please:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Run tests (pytest tests/ -v)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open a Pull Request

Changelog

v1.0.0 (2025-01-06)

Implemented (Phase 9/13 Complete):

✅ FastAPI REST API with OpenAPI documentation
✅ Celery async task processing with eventlet pool
✅ PostgreSQL database with 6 migrations
✅ Redis caching and message broker
✅ LLM provider abstraction (OpenRouter)
✅ External API integration (WeatherAPI, Google Places)
✅ Webhook system with HMAC signatures
✅ Comprehensive error handling and categorization
✅ Two-level caching system
✅ Rate limiting with token bucket algorithm
✅ Cost tracking and optimization
✅ Flower monitoring dashboard
✅ Docker deployment
✅ 333 unit and integration tests

Planned:

🚧 Phase 10: Next.js TypeScript client
🚧 Phase 11: Prometheus & Grafana monitoring
🚧 Phase 12: Streaming responses, multi-model support
🚧 Phase 13: Conditional workflows, parallel execution

Built with ❤️ by [Your Team]

Name		Name	Last commit message	Last commit date
.claude		.claude
app		app
cli		cli
migrations		migrations
prompts		prompts
tests		tests
workflows		workflows
.env.example		.env.example
.gitignore		.gitignore
.mcp.json		.mcp.json
.python-version		.python-version
API_GUIDE.md		API_GUIDE.md
CLAUDE.md		CLAUDE.md
CLI_GUIDE.md		CLI_GUIDE.md
DEPLOYMENT_GUIDE.md		DEPLOYMENT_GUIDE.md
Dockerfile		Dockerfile
README.md		README.md
TESTING_GUIDE.md		TESTING_GUIDE.md
WORKFLOW_SCHEMA.md		WORKFLOW_SCHEMA.md
debug_timeout.py		debug_timeout.py
docker-compose.yml		docker-compose.yml
monitor_job.bash		monitor_job.bash
requirements.txt		requirements.txt
test_concurrency.bash		test_concurrency.bash
test_llm_service.bash		test_llm_service.bash
test_webhook_signature.py		test_webhook_signature.py

Latest commit

History

Folders and files

Repository files navigation

LLM Processing Server

📋 Table of Contents

What is This?

The Problem It Solves

The Solution

Who Should Use This?

Key Features

✅ Core Capabilities

📊 Current Status

Quick Start

Prerequisites

5-Minute Setup

Installation

Docker Installation (Recommended)

Step 1: Install Docker

Step 2: Clone Repository

Step 3: Configure Environment

Step 4: Run Database Migrations

Step 5: Start Services

Step 6: Verify Installation

Local Development Setup

Deployment

Deploy to Render (Recommended)

Overview

Step 1: Create PostgreSQL Database

Step 2: Create Redis Instance

Step 3: Create Web Service (FastAPI API)

Step 4: Create Background Worker (Celery)

Step 5: Run Database Migrations

Step 6: Create Flower Monitoring (Optional)

Step 7: Verify Deployment

IPv6 Connectivity Issues

Vercel Deployment

Alternative Architectures for Vercel

Usage Guide

Basic Workflow Execution

1. Submit a Workflow

2. Check Job Status (Polling)

3. Webhook Delivery (Recommended)

List Jobs

Delete Jobs

API Reference

Authentication

Rate Limits

Workflow System

What is a Workflow?

Step Types

Variable Resolution

Creating Custom Workflows

Architecture

System Components

Technology Stack

Request Flow

Scaling Strategy

Monitoring & Debugging

Flower Dashboard

Health Checks

Logs

Database Queries

Advanced Topics

Custom Workflows

Environment-Specific Configuration

Cost Optimization

Troubleshooting

Common Issues

1. Redis Connection Refused

2. Database Connection Pool Exhausted

3. Event Loop Closed

4. Webhook Signature Validation Fails

5. IPv6 Connection to Supabase Fails

Debug Mode

Support & Resources

License

Contributing

Changelog

v1.0.0 (2025-01-06)