Mesh Audio Editor

Real-time mesh viewer and editor supporting conversion of meshes to rigid body audio models using Linear Modal Analysis/Synthesis.

Features

General

• Create and delete meshes and mesh instances
  • Editable mesh primitives (Rect, Circle, Cube, IcoSphere, UVSphere, Torus, Cylinder, Cone)
  • Load .obj and .ply mesh files (via tinyobjloader and tinyply)
• Select meshes, vertices, edges, or faces by clicking or box selection, with optional x-ray selection
• Flat/smooth/wireframe mesh rendering
• Translate, rotate, and nonuniformly scale meshes and instances with numeric inputs or a Blender-style transform gizmo
• Edit the camera with the mouse wheel, numeric inputs, or a Blender-style orientation gizmo
• Edit cameras and scene lighting
• Render face/vertex normals as lines for debugging
• Render bounding box wireframes for debugging
• Depth-aware silhouette outlines for active meshes and instances
• Infinite grid with horizon fade

Audio

• Enable/disable audio output and change device and native format/sample rate
• Volume / Mute
• Generate a physical audio model for any mesh; see Physical audio modeling
  • Click on an audio mesh to excite the nearest vertex, or trigger a selected vertex in the audio menu
  • Strike any number of objects and vertices concurrently (polyphonic modal synthesis)
  • Edit synth params (gain, fundamental frequency, decay scale, click level) in real-time
• Load RealImpact object meshes and microphone positions

Rendering and architecture

• Native Metal rendering through metal-cpp. Note: This used to be a Vulkan project. For reference, dae664bd0f2bbdfcf309ba7d000ce9890edbcb38 is the last commit SHA using Vulkan.
• Bindless Tier-2 argument-buffer access and GPU-address vertex pulling from contiguous arenas for mesh data (vertices, indices, attributes, selection state, etc.)
• Batched indirect indexed draws with GPU-written instance counts and visibility remaps
• Directly mapped CPU/GPU storage on unified-memory Apple Silicon
• GPU-accelerated mouse interactions
• Half-edge iterators for mesh topology operations
• Shader hot reloading: edit and recompile MSL at runtime from the UI
• Matching C++/MSL structs and function-constant indices generated from YAML

Physical audio modeling

MeshEditor generates physical audio models from meshes using linear modal analysis and synthesis. The original implementation was a Georgia Tech PHYS-6260 Computational Physics project documented in the final report and poster below.

The report and poster describe a previous Faust synthesis path. The current modal resonator bank supports concurrent sounding objects and physics collision impacts. The last commit using Faust code generation is 31a817dd.

Impact audio experiments

These examples compare synthesized modal impacts with RealImpact recordings and scanned meshes. Each synthesized impact injects a short wideband pulse at the selected vertex. See the blog post for embedded audio players.

The cylinders shown in the images represent recorded microphone positions, but all recordings come from a single microphone centered near the impacted object, and the modal audio model does not implement any audio wave radiation modeling. All modal audio samples are generated by extracting estimated surface vibrations, as if recorded from a contact microphone.

Object Name	Mesh	Real Impact Audio	Modal Impact Audio
Ceramic Koi Bowl		Impact	Modal
Ceramic Pitcher		Impact	Modal
Glass Cup		Impact	Modal
Iron Mortar		Impact	Modal
Iron Skillet		Impact	Modal
Plastic Scoop		Impact	Modal
Small Swan Ceramic		Impact	Modal

glTF viewer

MeshEditor is a glTF 2.0 viewer, editor, and exporter. glTF scene nodes are mapped to corresponding MeshEditor objects (meshes, armatures, cameras, lights, empty objects), with the scene parenting hierarchy mirroring the glTF node hierarchy. Khronos-ratified extensions that affect scene appearance are imported, rendered, and editable. Load and save preserve glTF data except for cases listed in tests/RoundtripTest.cpp. PBR BRDF/lighting equations are taken directly from the reference glTF-Sample-Renderer shaders.

Metal function constants specialize PBR variants for the current materials, lights, and image-based lighting. Feature-mask changes rebuild affected pipeline states from cached MSL libraries.

glTF supported extensions

✅ supported | 🟨 partial | ⬜ not supported

Extension	Status	Notes
KHR_mesh_quantization	✅	Handled in fastgltf parser/import path
EXT_mesh_gpu_instancing	✅	Imported into MeshEditor instances
KHR_lights_punctual	✅	Imported into MeshEditor light entities
KHR_texture_transform	✅
KHR_texture_basisu	✅	KTX2 transcoded via basis_universal. BC7/ETC2/RGBA32 target selected based on device support
EXT_texture_webp	✅	Decoded via libwebp. Edited textures re-encode to lossless WebP on export
KHR_materials_emissive_strength	✅
KHR_materials_unlit	✅
KHR_materials_specular	✅
KHR_materials_sheen	✅
KHR_materials_ior	✅
KHR_materials_dispersion	✅
KHR_materials_transmission	✅	"Real transmission" toggle in Viewport Shading: on samples a pre-rendered scene texture; off samples the IBL.
KHR_materials_diffuse_transmission	✅
KHR_materials_volume	✅
KHR_materials_clearcoat	✅
KHR_materials_anisotropy	✅
KHR_materials_iridescence	✅
KHR_materials_variants	✅
KHR_physics_rigid_bodies	✅	(not yet ratified)
KHR_implicit_shapes	🟨	(not yet ratified) WIP
KHR_node_visibility	✅	Load and roundtrip are conformant. However, hiding a parent in the app without also hiding its descendants is dropped, since glTF can't express "parent invisible, children visible."
KHR_animation_pointer	⬜	Too much complexity for now, will add when I get to arbitrary property animation.
EXT_lights_image_based	✅	Imported as Scene IBL when present
EXT_meshopt_compression	✅	Compressed bufferViews decoded via meshoptimizer at import; geometry re-emits uncompressed on export

Build & run

See production libraries for source ownership, dependencies and optimization settings.

Install dependencies

Requires Apple Silicon and the Xcode command-line tools.

$ git clone --recursive git@github.com:khiner/MeshEditor.git
$ brew install cmake llvm eigen
$ brew link llvm --force

Clone, clean, build, and run

$ git clone --recurse-submodules git@github.com:khiner/MeshEditor.git
$ cd MeshEditor
$ ./script/Clean # optionally clean first
$ ./script/Build [--release]
$ cd build && ./MeshEditor [file|--empty] [--quiet|-q] [--headless] [--play [seconds]] [--record path.mp4 [--fps N]] [--screenshot path.webp] [--render basename]

• file can be a .gltf, .glb, .obj, .ply, .state (scene snapshot), or .actions (replayed action log). No file loads the default scene.

• --empty starts with an empty scene instead of the default scene.

• --quiet / -q suppresses timer output. All of --play, --record, and --screenshot use the presentation look with material preview shading and hidden overlays. --play and --record run animation and physics; --screenshot captures the first frame.

• --play [seconds] starts playback. Optional [seconds] auto-exits after the given duration. See --record below for how the duration is interpreted.

• --record path.mp4 runs playback and writes the viewport as an H.264 .mp4 via a ffmpeg subprocess (must be on PATH). When a look-through camera is active, only the camera-frame sub-rect (the area inside the dimmed overlay) is recorded. Otherwise the full viewport is recorded.

• Motion blur uses fast velocity reconstruction by default. Choose Full sampling in the viewport controls, or --motion-blur N (1–64), to average complete shutter renders for changing visibility, reflections, and lighting. --motion-blur fast selects the default method. Both methods share the shutter setting and keep editor overlays sharp.

• --fps N sets the recording framerate (default 60).

• --screenshot path.webp writes a single image. The format is chosen by extension (.webp lossless, .png, .jpg/.jpeg), which is optional and defaults to .webp. The captured region matches --record. On its own it exits after writing; combined with --play [seconds] or --record it grabs the frame and keeps running.

• --render basename writes the scene's corpus artifacts under basename.* (used by ./script/Render — see Render corpus).

• --render-queue dir renders one scene per dir/*.job file (output basename followed by one command-line argument per line) in a single headless process, and parallel workers can safely share one queue. Used by ./script/Render; capture settings are parsed the same way as direct launches.

• --headless runs without a window: the viewport renders offscreen at a fixed 1280x800 (2x pixel density) extent, and any capture flags read it back. Without a capture flag it renders one frame and exits, and a duration-less --play exits after one timeline loop. With MESHEDITOR_VALIDATE_ACTIONS enabled, replay and snapshot validation also compare the complete UI rendered offscreen.

The flags can be combined freely, except --render excludes --record and --screenshot (it derives its own outputs). --render --play N caps the video at N seconds.

Timing: interactively, the sim runs at wall-clock rate. Recording samples the viewport at fps, so the file plays at the same rate as the in-app preview. --play N exits after N seconds — wall-clock when interactive, video-seconds when recording. Headless runs (and --render anywhere) are instead fixed-step and GPU-paced: one timeline frame per tick with every tick captured, so output is deterministic and --fps is unused.

Render corpus

render/ contains committed demo output for every scene in the corpus and mirrors the source layout. It covers the built-in scenes, res/examples/, root audio samples, and glTF samples under external/. Isolation audio samples under samples/test/ use script/AudioCorpus; redundant glTF format variants are skipped. Each scene leaf has a visual (lossless .webp for static scenes, plus one per material variant, .mp4 for animated ones - one timeline loop per animation clip, back-to-back), the .actions replay log, a .log of console output, and a run.sh that opens that scene in the app. A scene with sound objects renders its audio and muxes it into the .mp4.

Binary artifacts are stored in git-lfs; fetch them after cloning:

$ git lfs install
$ git lfs pull

Regenerate the corpus with ./script/Render. render/Benchmarks/Overlays/ adds fixed editor captures using the overlay benchmark scene and cases. Run script/Render --overlays-only to update just these captures (--no-build uses the existing executable). These leaves contain a lossless .webp and a run.sh that reproduces the capture headlessly; console logs stay local. Overlay cases share the corpus queue, worker pool, and retry handling.

Overlay cases	Coverage
axes	Grid/axis lines in front of and behind geometry-attached outlines
overview, orthographic	Dense linked/mirrored meshes, normals, bounds, lines, points, camera/light helpers, animated bones and bone X-ray; perspective and orthographic views
edit-vertex, edit-edge, edit-face, edit-xray	GPU box selection followed by an orbit to reveal selected/unselected elements, with and without X-ray
pick-1 through pick-4	Successive picks through overlapping opaque, mirrored and blended objects, a masked hole, and cycle wraparound

Scenes render headless (no window or display needed) and in parallel (JOBS sets the worker count, default 8). It needs the glTF submodules, plus ffmpeg on PATH for videos. Rendering is fixed-step (one tick per timeline frame) and GPU-paced at a fixed extent, so artifacts are deterministic: after regenerating, git status shows only scenes whose rendering actually changed.

Fidelity gate

git status gates a change that must not move a pixel. A change that moves pixels by a bounded amount (screen-error-bounded cluster LOD) needs a measurement instead:

$ ./script/RenderFidelity --app-args "FLAGS"

renders the whole corpus into render_candidates/ with FLAGS handed to every render worker, leaving the committed render/ tree alone, then reports per scene the share of pixels that moved, the largest and 99th-percentile per-channel difference, and the mean. It exits nonzero when a scene exceeds its budget in script/render_budgets.json, where the default budget is exact and a track relaxes a scene deliberately with the measured number and the reason.

./script/CompareRenders <candidate-tree> [reference-tree] runs the same comparison over any two render trees, writing the full report as JSON with --json. Omit the reference tree to judge against the corpus as committed at HEAD, read through git so a dirty working tree cannot become the reference. Videos are compared on decoded frames (16 sampled evenly per video by default, --video-frames 0 for all of them) plus decoded audio, so the driver's occasional one-LSB pixel no longer flips a whole file's hash.

Stack

• Metal via metal-cpp + AppKit + ImGui: Graphics + immediate-mode UI/UX
• Apple SIMD: Numeric vector/matrix types + math
• entt: Entity Component System (ECS) for an efficient and scalable mixin-style architectural pattern
• macOS Core Audio: HAL output and native audio-file I/O
• FastFEM: Tetrahedralization, finite-element assembly, and modal eigensolves
• fastgltf glTF 2.0 scene loading
• JoltPhysics: Rigid body physics
• basis_universal KTX2 texture transcoding (KHR_texture_basisu)
• libwebp WebP texture decoding and lossless snapshot/texture encoding (EXT_texture_webp)
• tinyobjloader and tinyply: .obj and .ply mesh loading
• lunasvg: Render SVG icons to bitmaps
• Accelerate: vDSP Fourier transforms behind the spectrum plots and FastFEM's BLAS/LAPACK/sparse kernels
• ImPlot: Plotting
• boost-ext/ut: Testing

Development

Build options

glTF sample submenus: glTF-Sample-Assets and glTF_Physics are git submodules under external/, populating File > glTF Samples and File > glTF_Physics Samples. These submenus are empty if the submodules aren't initialized:

$ git submodule update --init external/glTF-Sample-Assets external/glTF_Physics

RealImpact dataset: The external/RealImpact submodule has each object's scanned mesh, texture, and measurement metadata. Use File > Import > RealImpact and load external/RealImpact/dataset/<id>_<name>/preprocessed.

$ git submodule update --init external/RealImpact

The complete recorded RealImpact samples can be triggered interactively in the app by tapping on vertices (and compared with the generated modal audio model). The audio samples are 2.8 GB per object, so they're not stored in git and need to be downloaded:

$ external/RealImpact/dataset/download.sh # every object, 128 GB
$ external/RealImpact/dataset/download.sh 9_BowlCeramic 22_Cup # download just these two

An object without a recordings still loads - the mesh, microphone positions, and impact vertices all appear, but no sample-based audio model is generated.

Quiet mode: Disable timer output at compile time (equivalent to always passing --quiet):

$ cmake -B build -DQUIET=ON .

Renderer benchmarks

Generate the local benchmark scenes, then run the suite:

$ script/GenerateRenderBenchmarks
$ script/Bench run

The large generated datasets under res/benchmarks/ are ignored by git; the small Overlays/ scene is committed for the render corpus. These scenes populate File > Benchmarks. Use script/GenerateRenderBenchmarks --profile smoke for a quick tooling check, script/Bench list to list cases, and script/Bench compare before.json after.json to compare runs.

The overlay corpus and performance checks share the same scene and case settings:

$ script/Bench run --scene-root res/benchmarks/Overlays
$ script/GenerateRenderBenchmarks --profile overlays --check

--profile overlays regenerates only the committed overlay scene, preserving the large local datasets. It contains 192 linked sphere instances (786,432 logical triangles) plus the small coverage fixtures.

Tests

./script/Build --test builds the glTF roundtrip suite, which needs the external/glTF-Sample-Assets and external/glTF_Physics submodules (see above).

$ ./script/Build --test
$ ./build/tests/MeshEditorTests

cmake --build build builds the rest of the tests:

Target	Covers
MeshEditorTests	glTF roundtrip
MeshEditorActionSerializeTest	Every action alternative through the action log
MeshEditorContactModelTest	Hertz contact time, effective mass, inertia decomposition
MeshEditorModalRenderTest	Superposition, thread independence, and the click's rate independence
MeshEditorCompressTest	.project archive round trip
MeshEditorModalSolveIntegrationTest	FastFEM surface solves, modal-component conversion, warm-start retention, and material rescaling

SURFACE_AUDIO=1 builds the surface-contact audio model (src/audio/surface/) and its own tests, benches and scores under tests/surface/. See src/audio/SurfaceContact.h for the entry points the core modal path reaches it through.

$ SURFACE_AUDIO=1 script/Build

VALIDATE_ACTIONS=1 builds the app to run File->[Debug] Roundtrip after every committed action: the log replays into a fresh session and the scene saves, clears and restores, aborting on the first divergence. Validation compares canonical state, the complete viewport texture, and the composed UI at the captured timeline position. An unset VALIDATE_ACTIONS (or 0) disables it on the next script/Build; script/Render builds with the same rule. --no-build retains the existing binary’s setting.

$ VALIDATE_ACTIONS=1 script/Build

FastFEM owns the analytical, tetrahedralization, real-mesh, and performance corpora documented in lib/FastFEM/README.md.

Update submodules

Submodules live in lib (libraries) and external. Here is my process for updating to the tip of all the lib submodule branches:

$ git submodule update --remote {path}/{submodule}
$ git add .
$ git cm -m "Update {submodule} ..."