Skip to main content

Curie 3D Reconstruction Pipeline

The Curie 3D Reconstruction Pipeline is a sophisticated system that transforms 2D product images into production-ready 3D models. Built around ComfyUI with custom ML nodes, it automates the entire reconstruction workflow from image input to textured 3D meshes suitable for e-commerce applications.

Overview​

This pipeline represents the core technology that powers Curie's 3D product visualization capabilities. It combines multiple state-of-the-art computer vision and machine learning techniques to create high-quality 3D models from photographs.

Key Capabilities​

  • Automated 3D Reconstruction: Convert 2D product images to 3D meshes
  • Multi-Stage Pipeline: Depth estimation, mesh generation, texturing, and optimization
  • Quality Assurance: Built-in metrics (SSIM, MSE) and validation systems
  • Production Scale: Handles batch processing with cloud deployment
  • Flexible Workflows: Configurable pipeline stages for different product types

Architecture​

System Components​

The reconstruction system is organized into several key concepts:

1. Nodes​

Python functions wrapped with metadata for ComfyUI integration. These handle specific tasks like:

  • Depth Estimation: MoGe, Depth-Anything models for depth map generation
  • Mesh Processing: COLMAP integration, mesh optimization, texture mapping
  • Alignment: Camera pose estimation and mesh alignment
  • Quality Control: Metric calculation and validation

2. Workflows (Prompts)​

JSON-defined directed graphs that connect nodes to form complete processing pipelines. Examples:

  • shoe_reconstruction.json: Complete shoe reconstruction workflow
  • hunyuan_test.json: Experimental workflows with new models
  • gather_ref_images.json: Reference image collection workflow

3. Pipelines​

High-level orchestration that manages workflow execution, artifact caching, and cloud integration:

  • Stage-by-stage execution with S3 artifact management
  • Automatic caching to avoid redundant computation
  • Integration with external systems (RunPod, AWS services)

Technology Stack​

Core Framework

  • ComfyUI: Workflow orchestration and node execution engine
  • Python 3.8+: Primary development language
  • PyTorch: Deep learning model inference
  • Docker: Containerized deployment and environment management

Computer Vision & 3D Processing

  • COLMAP: Structure from Motion (SfM) and Multi-View Stereo (MVS)
  • PyTorch3D: 3D deep learning operations and mesh processing
  • Open3D: Point cloud and mesh manipulation
  • OpenCV: Image processing and computer vision utilities
  • Trimesh: Mesh processing and analysis

Machine Learning Models

  • MoGe: Monocular geometry estimation
  • Depth-Anything V2: Robust depth estimation
  • Hunyuan3D: Multi-view 3D reconstruction
  • DIS (Background Removal): Image segmentation
  • FBCNN: Image denoising and enhancement

Infrastructure

  • AWS S3: Model storage and artifact management
  • RunPod: GPU-accelerated cloud execution
  • DynamoDB: Metadata and job tracking
  • Redis: Caching and session management

Project Structure​

curie-reconstruction/
├── src/curie/
│   ├── nodes/                  # Custom ComfyUI nodes
│   │   ├── alignment_nodes.py  # Camera alignment and pose estimation
│   │   ├── depth_nodes.py      # Depth map generation
│   │   ├── mesh_nodes.py       # 3D mesh processing
│   │   ├── image_nodes.py      # Image preprocessing
│   │   └── hunyuan_nodes.py    # Hunyuan3D integration
│   ├── tools/                  # Utility scripts and tools
│   └── curie_util.py          # Common utilities
├── workflows/                  # ComfyUI workflow definitions
│   ├── shoe_reconstruction/    # Production shoe workflows
│   └── experimental/           # Experimental workflows
├── pipelines/                  # High-level pipeline definitions
│   ├── shoe_reconstruction.json
│   └── gather_ref_images.json
├── deps/                      # External dependencies
│   ├── ComfyUI/              # Curie's ComfyUI fork
│   ├── colmap/               # COLMAP installation
│   ├── dust3r/               # DUSt3R model
│   └── Hunyuan3D-2/          # Hunyuan3D implementation
├── sample_data/               # Test datasets
└── scripts/                  # Installation and utility scripts

Installation & Setup​

Prerequisites​

  • CUDA-capable GPU with 8GB+ VRAM (recommended: RTX 3080 or better)
  • Ubuntu 20.04+ or compatible Linux distribution
  • Docker and Docker Compose
  • Git with LFS support
  • Python 3.8+ with conda/mamba

Quick Start with Docker​

  1. Clone the repository

    git clone https://github.com/curievision/curie-reconstruction.git
    cd curie-reconstruction

  2. Build Docker image

    docker build -f Dockerfile.app -t curie:reconstruction .

  3. Run with GPU support

    docker-compose up

  4. Access ComfyUI interface

    • Navigate to http://localhost:8189
    • Load workflows from the workflows/ directory

Manual Installation​

For development or custom installations:

# Install system dependencies
./scripts/install/0_install_cuda.sh
./scripts/install/1_install_deps.sh

# Install Python environment
./scripts/install/2_install_miniconda.sh
conda activate curie

# Build dependencies
./scripts/install/3_build_colmap.sh
./scripts/install/4_build_mve_mvs.sh

# Install Python packages
pip install -r requirements.txt
pip install -e .

Reconstruction Workflows​

Standard Shoe Reconstruction​

The primary production workflow for footwear:

  1. Image Preprocessing

    • Background removal with DIS model
    • Image denoising and enhancement
    • Resolution normalization

  2. Depth Estimation

    • MoGe or Depth-Anything model inference
    • Multi-view depth map generation
    • Depth map refinement and filtering

  3. 3D Reconstruction

    • Point cloud generation from depth maps
    • COLMAP Structure from Motion
    • Dense reconstruction with MVS

  4. Mesh Processing

    • Poisson surface reconstruction
    • Mesh decimation and optimization
    • UV mapping and texture generation

  5. Quality Assurance

    • SSIM and MSE metric calculation
    • Visual quality assessment
    • Production readiness validation

Pipeline Configuration​

// Example pipeline stage configuration
{
  "stage_name": "depth_estimation",
  "workflow": "workflows/depth_estimation.json",
  "inputs": {
    "image_path": "${previous_stage.output_images}",
    "model_name": "MoGe_ViT_L_fp16"
  },
  "outputs": {
    "depth_maps": "depth/*.png",
    "point_clouds": "pointclouds/*.ply"
  },
  "cache_key": "depth_v2.1"
}

Custom Nodes​

Depth Estimation Nodes​

@ComfyNode(CAT_DEPTH, return_names=["depth", "output_path"])
def process_moge(
    model: MoGeModel,
    image: ImageTensor,
    resolution_level: int = 9,
    remove_edge: bool = True,
    output_format: str = "glb"
) -> tuple[DepthMap, str]:
    """Generate depth maps using MoGe model"""
    # Model inference and processing logic

Mesh Processing Nodes​

@ComfyNode(CAT_MESH, return_names=["rendered", "mask", "depth_map"])
def render_mesh(
    pytorch3d_mesh: Pytorch3dMesh,
    R: Matrix,
    T: Matrix,
    fov: float = 60,
    render_width: int = 512,
    render_height: int = 512
) -> tuple[ImageTensor, MaskTensor, DepthMap]:
    """Render 3D mesh from specified viewpoints"""
    # Rendering logic with PyTorch3D

Cloud Deployment​

RunPod Integration​

The pipeline is designed for cloud execution on RunPod's GPU infrastructure:

# Deploy to RunPod
python src/curie/tools/run_serverless_pipeline.py \
  --pipeline shoe_reconstruction \
  --variables "product_name:Nike-Air-Jordan-1"

Pipeline Orchestration​

# Pipeline execution with automatic caching
orchestrator = PipelineOrchestrator(
    pipeline="shoe_reconstruction",
    frontend="http://127.0.0.1:8080", 
    variables="product_name:example-shoe",
    enable_cache=True,
    metadata=run_metadata
)

success = orchestrator.execute_pipeline()

AWS Integration​

  • S3 Storage: Automatic artifact upload and retrieval
  • DynamoDB: Job metadata and status tracking
  • CloudWatch: Comprehensive logging and monitoring

Quality Metrics​

Automated Quality Assessment​

The pipeline includes built-in quality metrics:

# SSIM (Structural Similarity Index)
ssim_score = calculate_ssim(rendered_image, reference_image)

# MSE (Mean Squared Error)  
mse_score = calculate_mse(predicted_depth, ground_truth_depth)

# Production readiness threshold
is_production_ready = ssim_score > 0.85 and mse_score < 0.1

Visual Quality Control​

  • Reference image alignment: Ensure proper camera pose estimation
  • Mesh topology validation: Check for holes, non-manifold geometry
  • Texture quality assessment: Verify UV mapping and material properties

Performance Optimization​

Caching Strategy​

The pipeline implements multi-level caching:

  1. S3 Artifact Caching: Avoid re-computing expensive operations
  2. Joblib Node Caching: Cache individual node outputs
  3. Model Weight Caching: Reuse loaded models across runs

GPU Optimization​

  • Mixed precision inference for faster model execution
  • Dynamic batching for multi-image processing
  • Memory management with automatic garbage collection

Scalability​

  • Horizontal scaling with multiple RunPod instances
  • Load balancing across available GPU resources
  • Batch processing for multiple products simultaneously

Development & Testing​

Local Development​

# Start ComfyUI development server
cd deps/ComfyUI
python main.py --listen 0.0.0.0 --port 8189

# Load custom nodes
ln -s /path/to/curie/src/curie/nodes /path/to/ComfyUI/custom_nodes/curie

Testing Framework​

# Run unit tests
python -m pytest tests/

# Test specific workflow
python src/curie/tools/test_workflow.py \
  --workflow workflows/shoe_reconstruction/main.json \
  --input sample_data/shoe/

Adding Custom Nodes​

from easy_nodes import ComfyNode
from curie.nodes.common import CAT_MESH

@ComfyNode(CAT_MESH, return_names=["processed_mesh"])
def custom_mesh_processor(
    mesh: Pytorch3dMesh,
    smoothing_factor: float = 0.5
) -> Pytorch3dMesh:
    """Custom mesh processing logic"""
    # Implementation here
    return processed_mesh

Monitoring & Debugging​

CloudWatch Integration​

All pipeline runs are logged to AWS CloudWatch with:

  • Structured logging with JSON format
  • Performance metrics (processing time, memory usage)
  • Error tracking with full stack traces
  • Resource utilization monitoring

Debugging Tools​

# View pipeline logs
python src/curie/tools/view_logs.py --run-id <run_id>

# Analyze performance metrics
python src/curie/tools/analyze_performance.py --product <product_name>

# Debug failed reconstructions
python src/curie/tools/debug_reconstruction.py --workflow-id <workflow_id>

API Reference​

Pipeline Execution API​

from curie.tools.pipeline_orchestrator import PipelineOrchestrator

# Execute reconstruction pipeline
orchestrator = PipelineOrchestrator(
    pipeline="shoe_reconstruction",
    variables="product_name:example-product,variant:black",
    enable_cache=True
)

result = orchestrator.execute_pipeline()
if result.success:
    model_url = result.final_artifacts['textured_mesh.glb']

Node Development API​

from curie.nodes.common import ComfyNode, CAT_MESH
from easy_nodes import ImageTensor, NumberInput

@ComfyNode(CAT_MESH)  
def my_custom_node(
    input_image: ImageTensor,
    strength: float = NumberInput(1.0, 0.0, 2.0, 0.1)
) -> ImageTensor:
    """Custom node implementation"""
    # Processing logic
    return output_image

Contributing​

Development Workflow​

  1. Fork and clone the repository
  2. Set up development environment with Docker or manual installation
  3. Create feature branch for your changes
  4. Add tests for new functionality
  5. Submit pull request with detailed description

Adding New Models​

To integrate a new ML model:

  1. Create model wrapper in appropriate category
  2. Implement ComfyUI node with proper type annotations
  3. Add workflow examples demonstrating usage
  4. Update documentation and tests

Code Standards​

  • Python PEP 8 styling with black formatter
  • Type hints for all function signatures
  • Comprehensive docstrings for public APIs
  • Unit tests for critical functionality

Ready to start reconstructing? This documentation provides everything you need to set up and run the 3D reconstruction pipeline.