Display a 3D model alongside 2D images and audiovisual content within a single viewing experience

[JulieWinchester]

As a biology educator

I want to display a 3D mesh of a skull alongside a 2D image of the actual skull and a video scrolling through a CT scan of that skull.

So that I can provide context for the 3D model, showing its original source and creation process

copy of IIIF/iiif-3d-stories#11

[vincentmarchetti]

Some examples using X3D standard and web-based viewers
http://www.kshell.com/pages/web3d201811/visibility/index.html -- showing detailed alignment of a 3D model with displayed 2D draft drawings

Less technically oriented, but showing 3D glTF models, video, and audio in a single scene: -- scene has car, plane, video of tv commercial & USAF stock footage, and audio of song "Rocket 88"
https://spri-open-resources.s3.us-east-2.amazonaws.com/scene88/index_x3dom.html
https://spri-open-resources.s3.us-east-2.amazonaws.com/scene88/index_xite.html

Finally, don't have an example yet, but X3D could probably support the questioner's use case, along with volumetric rendering of 3D CT/MRI scan data, in a single scene.

[JulieWinchester]

During the IIIF Annual Conference 2023 the week of June 5 2023, there was discussion of potential requirements and attributes for annotating canvases or 2D content resources within 3D scenes:

• Needs a transformation position target to place the canvas into the scene, painting a canvas centroid "origin" point into the scene at the position target
• When a canvas is placed into a 3D scene, the "center" of the canvas for 3D purposes should be considered the center of the 2D resource, NOT the top-left 2D origin. In other words, if a canvas with height and width of 200, 400 is placed at the 3D coordinate origin 0, 0, 0, then the 2D canvas point to be placed at the 3D coordinates 0, 0, 0 would be 2D canvas coordinates 100, 200, not 2D canvas coordinates 0, 0.
• How do 2D pixels relate to 3D coordinate units, since the two will often exist at wildly different scales? Potentially we could include a scale factor property. Alternately instead of (or in addition to) using a single position transformation point and placing the canvas at that point, the target selector could specify a 2D rectangle area in space to which the canvas would be transformed

Not discussed during the conference, but we should consider whether 2D canvas resources need a normal property of some kind?