Matrix Eyes

Matrix Eyes is a Rust port of Apple Depth Pro project to convert a photo image into an autostereogram or 3D mesh.

For running ML models, the burn library is used. There's also an experimental version using candle in the candle tag.

The python tag contains an older Python-based version which supported both MiDaS and Apple Depth Pro depth estimation algorithms.

This app reuses some code from Cybervision, and tries to achieve the same goal. Cybervision uses a "classic" structure-from-motion approach and reconstructs objects from multiple views, while Matrix Eyes uses a pretrained machine learning model to add depth to a single image.

Examples

Image 1

Source image:

Depth data extracted by Matrix Eyes:

Generated stereogram - works best when viewed in fullscreen:

Instructions

Installation

Download a copy of Matrix Eyes from Releases and extract it into in a directory. The following versions are available:

• Windows
  • fusion (fusion enabled, runs on a GPU with 8GB of VRAM, supports vendor-neutral GPU and CUDA acceleration)
  • fusion-autotune (fusion and autotune enabled, runs on a GPU with 8GB of VRAM, supports vendor-neutral GPU and CUDA acceleration)
• Ubuntu
  • fusion-autotune (fusion and autotune enabled, runs on a GPU with 8GB of VRAM, supports vendor-neutral GPU and CUDA acceleration)
  • fusion (fusion enabled, runs on a GPU with 8GB of VRAM, supports vendor-neutral GPU and CUDA acceleration)
• macOS
  • fusion-autotune (wgpu version with fusion and autotune enabled, uses 4 GB when running)
  • fusion (wgpu version with fusion with fusion enabled, uses 3 GB when running)

To use CUDA acceleration in Windows, download the Burn CUDA libraries artifact and extract its contents into the same directory. Set the CUDA_PATH environment variable to the path where you've extracted matrix-eyes and the Burn CUDA libraries artifact.

Download the model checkpoints:

mkdir checkpoints
curl -LJ -o checkpoints/depth_pro.pt https://ml-site.cdn-apple.com/models/depth-pro/depth_pro.pt

Usage

To gerate a depth image, run:

matrix-eyes [--focal-length=<focal-length>] [--checkpoint-path=<checkpoint-path>] [--image-output-format=<depthmap|stereogram>] [--resize-scale=<scale>] [--stereo-amplitude=<amplitude>] [--mesh=<plain|vertex-colors|texture-coordinates>] [--convert-checkpoints] [--acceleration=<wgpu|cuda|cpu>] [--precision=<f16|f32>] <source> <output>

--focal-length=<focal-length> is an optional argument to specify a custom focal length for images with perspective projection, for example, --focal-length=26; this should be the image's focal length in 35mm equivalent. If not specified, EXIF metadata will be used; if EXIF data is not available, the focal length will be estimated using Depth Pro.

--checkpoint-path=<checkpoint-path> is an optional argument to specify a custom path to the Depth Pro checkpoints file, --checkpoint-path=./ckpoint.pt.

--image-output-format=<depthmap|stereogram> is an optional argument to specify the image output format, for example --image-output-format=depthmap or --image-output-format=stereogram. depthmap (the default option) outputs a depth map image, while --image-output-format=stereogram outputs a stereogram image.

--resize-scale=<scale> is an optional argument to specify a custom scale for the stereogram image output, for example --resize-scale=0.25. This can help with making noise pixels large enough to be visible.

--stereo-amplitude is an optional argument to specify the maximum offset/depth for stereograms (relative to image width); might need to be reduced if most of the image consists of foreground objects, for example --stereo-amplitude=0.0625.

--mesh=<plain|vertex-colors|texture-coordinates> is an optional argument to specify how to output OBJ and PLY meshes mode, for example --mesh=vertex-colors or --mesh=texture-coordinates. plain (the default option) outputs the mesh without any color or texture, vertex-colors outputs the mesh with colors assigned to every vertex, and texture-coordinates will add texture coordinates.

--convert-checkpoints will convert checkpoints from a .pt (Python pickle) format into a more efficient Burn format.

--acceleration=<wgpu|cuda|cpu> is an optional argument to specify which hardware/acceleration framework to use, for example --acceleration=wgpu. cuda only works in Linux and Windows versions and requires a CUDA-compatible GPU, while wgpu and cpu run on all platforms.

--precision=<f16|f32> is an optional argument to specify floating point precision, for example --precision=f32. If not specified, f16 precision will be used, as it uses less VRAM and produces results almost identical to f32.

<source> specifies the filename for the source file; supported formats are jpg and png.

<output> is the output filename:

• If the filename ends with .obj, this will save a 3D Wavefront OBJ file.
• If the filename ends with .ply, this will save a 3D PLY binary file.
• If the filename ends with .png, this will save a PNG image (depth map or stereogram).
• If the filename ends with .jpg, this will save a JPEG image (depth map or stereogram).

GPU details

Matrix Eyes was tested to support CPU-only and GPU-accelerated processing on:

• Apple Macbook Pro M4 Max (2024) (wgpu version)
• Apple Macbook Pro M1 Max (2021) (wgpu versions)
• Windows 11, i7-11800H, Geforce RTX 3070 (cuda and wgpu versions)
  • f32 precision needs more than 8GB VRAM and causes exceptions or device loss issues.
• Fedora 44, i7-11800H, Geforce RTX 3070 (cuda and wgpu versions)
  • f32 precision needs more than 8GB VRAM and causes exceptions or device loss issues.