.dtdata sidecar for per-pixel edit data, first used by external raster masks

README.md

@@ -238,6 +238,7 @@ Required dependencies (minimum version):
 * Exiv2 0.27.2 *(but at least 0.27.4 built with ISO BMFF support needed for Canon CR3 raw import)*
 * potrace 1.16
 * pugixml 1.8
+* libarchive 3.8.5 *(for the `.dtdata` sidecar; also used for AI models download)*
 
 Required dependencies (no version requirement):
 * Lensfun *(for automatic lens correction)* (Note: alpha 0.3.95 and git master branch are not supported)
@@ -255,7 +256,6 @@ Optional dependencies (minimum version):
 * Imath 3.1.0 *(for 16-bit "half" float TIFF export and faster import)*
 * libavif 0.9.3 *(for AVIF import & export)*
 * ONNX Runtime 1.18 *(for AI inference)*
-* libarchive 3.8.5 *(for AI models download)*
 * libheif 1.13.0 *(for HEIF import & export; also for AVIF import if no libavif)*
 * libjxl 0.7.0 *(for JPEG XL import & export)*
 * WebP 0.3.0 *(for WebP import & export)*

RELEASE_NOTES.md

@@ -155,6 +155,18 @@ changes (where available).
 
 ## Other Changes
 
+- A new `.dtdata` sidecar next to the XMP holds per-pixel edit data
+  such as raster masks. It is only created for images that use such
+  data, and only when sidecar writing is enabled; most images never get
+  one. It follows the image when it is moved, renamed, duplicated or
+  deleted.
+
+- The external raster masks module imports its mask into the image's
+  `.dtdata` sidecar instead of referencing a file in a root folder.
+  Each instance holds one imported mask. Existing edits that point at
+  a folder keep working, and the folder chooser remains available when
+  sidecar writing is set to "never".
+
 - Added a new collection filter for the original image dimensions.
 
 - Support for Canon's Highlight Tone Priority.
@@ -382,7 +394,8 @@ changes (where available).
 
 ### Mandatory
 
-- N/A
+- libarchive, previously only needed for AI model downloads, is now
+  required for the `.dtdata` sidecar
 
 ### Optional
 

dev-doc/README.md

@@ -31,6 +31,11 @@ This guide covers building Image Operation (IOP) modules for darktable's darkroo
 | **[AI.md](AI.md)** | AI subsystem architecture, backend API, and how to add new AI features |
 | **[AI_Tasks.md](AI_Tasks.md)** | Reference for each AI task: mask, denoise, upscale |
 
+### Sidecar Files
+| File | Description |
+|------|-------------|
+| **[dtdata_sidecar.md](dtdata_sidecar.md)** | The `.dtdata` sidecar for raster masks and other per-pixel edit data |
+
 ### Guides
 | File | Description |
 |------|-------------|

dev-doc/dtdata_sidecar.md

@@ -0,0 +1,115 @@
+# The .dtdata sidecar
+
+Per-pixel data that belongs to an edit but does not fit an XMP: raster
+masks today, depth maps and painted layers later. Design discussion:
+https://github.com/darktable-org/darktable/issues/22226
+
+## File
+
+`IMG_0001.RAF.dtdata` next to `IMG_0001.RAF.xmp`; duplicates get the same
+`_NN` suffix as their XMP. A flat zip with no manifest:
+
+```
+version                 "1"
+mask-<sha1>.png         8-bit gray, an alpha mask
+depth-<sha1>.png        16-bit gray
+brush-<sha1>.png        8-bit gray
+patch-<sha1>.png        pixels
+```
+
+An entry is named by the SHA-1 of its bytes and is never modified. A new
+mask is a new entry; history steps point at different ones. Readers
+ignore entries they do not know, writers keep entries they cannot
+attribute, so an older darktable never drops what a newer one wrote.
+Nothing in the file describes an entry: that lives with the reference.
+
+## Reference
+
+`dt_dtdata_ref_t` in `src/common/dtdata.h` is the only thing stored in
+history: entry name, kind, origin (regenerable by a model or made by a
+person), size, bit depth, producer. It has no path. The entry is looked
+up in the sidecar beside the XMP the history came from, by the same rule
+that finds the XMP, so moving a folder breaks nothing. Like the XMP, the
+sidecar sits beside the original when it is reachable and beside the
+local copy otherwise. Creating a local copy copies the sidecar into the
+cache with it, so masks imported while the original is offline land in
+the cache sidecar and earlier ones stay readable; a read that finds no
+entry beside the original falls back to the cache sidecar while the
+image has a local copy. Resetting the local copy merges the cache
+sidecar back into the original's and deletes it; if the merge fails the
+local copy is kept whole for a retry. On read the SHA-1 is
+checked against the name; a missing or damaged entry is an empty mask
+and a log line.
+
+The reference is copied verbatim wherever module params travel. The
+first consumer, `src/iop/rasterfile.c`, lays the fields out inline in its
+params so introspection can see them.
+
+## API
+
+`src/common/dtdata.c`, libarchive for the zip and libpng in memory for
+the entries:
+
+- `dt_dtdata_enabled()`: FALSE when sidecar writing is "never". No new
+  entry is written then and there is no database fallback; existing
+  sidecars are still read, and sweeps and merges still rewrite or delete
+  them. The rasterfile module falls back to its previous behavior in
+  that case, a file referenced by path inside the raster mask root
+  folder
+- `dt_dtdata_path()`, `dt_dtdata_path_for_image()`
+- `dt_dtdata_write_gray()`, `dt_dtdata_read_gray()`, `dt_dtdata_list_entries()`
+- `dt_dtdata_merge()`: copy one image's entries into another's; called
+  from `dt_history_copy_and_paste_on_image()` because a pasted reference
+  needs its entry on the destination
+- `dt_dtdata_register_scanner()`, `dt_dtdata_sweep()`:
+  cleanup, see below
+- `dt_dtdata_find_all()`: the sidecars of an image and its duplicates by
+  name pattern, for the delete job, since an xmp may not exist yet
+- `dt_dtdata_file_*`: the same on explicit paths, used by the cmocka test
+  `src/tests/unittests/common/test_dtdata.c`
+
+Writers rewrite the whole zip to a temp file and rename it, holding the
+write side of a read-write lock; readers open the file per call under
+the read side, since Windows refuses the rename while the file is open.
+A zip that cannot be read to its end is never rewritten, swept or merged
+from. The rasterfile module caches the decoded mask by its params hash,
+and the SHA-1 in the entry name is part of those params, so the cache is
+content-addressed for free.
+
+## Cleanup
+
+An entry is garbage when no row of the image's stored history references
+it. That is decided only when the history is final: undo lives inside one
+darkroom session and older history items can be clicked at any time, so
+reset, instance delete and re-import never touch the file. The sweep runs
+when the darkroom leaves or switches image, right after the history is
+written, and from the lighttable "compress history" job, which has no
+undo. "discard history" leaves the file alone, since the lighttable undo
+can bring the history back; the sweep removes the orphaned entries, and
+the file once it is empty, the next time the image is left in the darkroom
+or compressed.
+
+Modules are plugins, so core code cannot read their params. A producer
+registers a scanner from `init_global()`, "given a params blob and its
+version, which entry does it reference", together with the entry kinds it
+writes. The sweep only removes entries of a registered kind, so a build
+that lacks the producer of some kind leaves those entries alone.
+`rasterfile` registers for the `mask` kind.
+
+A scanner returning -1 cancels the sweep for that image, but that only
+covers rows that reach the sweep unchanged. Known limitation: history
+loading (`src/develop/develop.c`) replaces a row whose params version is
+newer than the module knows with default params before any sweep runs,
+as it does for every module, so an older build sweeps away masks that
+only newer-version rows referenced.
+
+## Lifecycle
+
+The sidecar follows the XMP in `dt_image_rename()` (move and rename; a
+sidecar that cannot be moved is copied, since its masks exist nowhere
+else), both branches of the delete job, and local-copy reset. It is refused as
+an importable file. Copies and duplicates get their entries through the
+history paste. Not handled yet: the "write sidecar files" action (the
+file is written at import time, not synced from the database), the
+crawler, XMP-only transfer, and styles carrying a reference to an entry
+the target image does not have.

packaging/macosx/BUILD-ARM64.txt

@@ -4,7 +4,7 @@ How to make disk image with darktable application bundle (64 bit ARM only):
     They will need some tuning, so before you install anything add this line to /opt/local/etc/macports/variants.conf:
      +no_gnome +no_x11 +quartz -x11 -gnome -gfortran
     Install required dependencies:
-     $ sudo port install exiv2 libgphoto2 gtk-osx-application-gtk3 lensfun librsvg openexr json-glib GraphicsMagick openjpeg webp libsecret pugixml osm-gps-map adwaita-icon-theme intltool iso-codes libomp gmic-lib libheif portmidi libsdl2 libavif libjxl potrace
+     $ sudo port install exiv2 libgphoto2 gtk-osx-application-gtk3 lensfun librsvg openexr json-glib GraphicsMagick openjpeg webp libsecret pugixml osm-gps-map adwaita-icon-theme intltool iso-codes libomp gmic-lib libheif portmidi libsdl2 libavif libjxl potrace libarchive
     Clone darktable git repository (in this example into ~/src):
      $ mkdir ~/src
      $ cd ~/src

packaging/macosx/BUILD.txt

@@ -14,7 +14,7 @@ How to make disk image with darktable application bundle (64 bit Intel only):
      $ curl -Lo ~/ports/x11/pango/files/patch-meson-examples-tests.diff https://github.com/macports/macports-ports/raw/26241fac142ac2bbe2a9071918ff20b301c66f4b/x11/pango/files/patch-meson-examples-tests.diff
      $ portindex ~/ports
 
-     $ sudo port install exiv2 libgphoto2 gtk-osx-application-gtk3 lensfun librsvg openexr json-glib GraphicsMagick openjpeg webp libsecret pugixml osm-gps-map adwaita-icon-theme intltool iso-codes libomp gmic-lib libheif portmidi libsdl2 libavif libjxl potrace
+     $ sudo port install exiv2 libgphoto2 gtk-osx-application-gtk3 lensfun librsvg openexr json-glib GraphicsMagick openjpeg webp libsecret pugixml osm-gps-map adwaita-icon-theme intltool iso-codes libomp gmic-lib libheif portmidi libsdl2 libavif libjxl potrace libarchive
     Clone darktable git repository (in this example into ~/src):
      $ mkdir ~/src
      $ cd ~/src

packaging/nix/flake.nix

@@ -62,6 +62,7 @@
                 lensfun
                 lerc
                 libaom
+                libarchive
                 #libavif # TODO re-enable once cmake files are fixed (#425306)
                 libdatrie
                 libepoxy

src/CMakeLists.txt

@@ -60,6 +60,7 @@ FILE(GLOB SOURCE_FILES
   "common/histogram.c"
   "common/history.c"
   "common/history_snapshot.c"
+  "common/dtdata.c"
   "common/image.c"
   "common/image_cache.c"
   "common/imagebuf.c"
@@ -522,29 +523,27 @@ endif()
 find_package(CURL 7.56 REQUIRED)
 list(APPEND LIBS CURL::libcurl)
 
-if(USE_AI)
-  # libarchive for AI model ZIP extraction
-  # On macOS with Homebrew, libarchive is keg-only
-  if(APPLE)
-    execute_process(
-      COMMAND brew --prefix libarchive
-      OUTPUT_VARIABLE HOMEBREW_LIBARCHIVE_PREFIX
-      OUTPUT_STRIP_TRAILING_WHITESPACE
-      ERROR_QUIET
-      RESULT_VARIABLE HOMEBREW_LIBARCHIVE_RESULT
-    )
-    if(HOMEBREW_LIBARCHIVE_RESULT EQUAL 0 AND EXISTS "${HOMEBREW_LIBARCHIVE_PREFIX}")
-      list(APPEND CMAKE_PREFIX_PATH "${HOMEBREW_LIBARCHIVE_PREFIX}")
-    endif()
-  endif()
-  find_package(LibArchive)
-  if(NOT LibArchive_FOUND)
-    message(FATAL_ERROR "libarchive not found")
-  else()
-    include_directories(SYSTEM ${LibArchive_INCLUDE_DIRS})
-    list(APPEND LIBS ${LibArchive_LIBRARIES})
+# libarchive for the .dtdata sidecar (common/dtdata.c) and AI model ZIP extraction
+# On macOS with Homebrew, libarchive is keg-only
+if(APPLE)
+  execute_process(
+    COMMAND brew --prefix libarchive
+    OUTPUT_VARIABLE HOMEBREW_LIBARCHIVE_PREFIX
+    OUTPUT_STRIP_TRAILING_WHITESPACE
+    ERROR_QUIET
+    RESULT_VARIABLE HOMEBREW_LIBARCHIVE_RESULT
+  )
+  if(HOMEBREW_LIBARCHIVE_RESULT EQUAL 0 AND EXISTS "${HOMEBREW_LIBARCHIVE_PREFIX}")
+    list(APPEND CMAKE_PREFIX_PATH "${HOMEBREW_LIBARCHIVE_PREFIX}")
   endif()
-endif(USE_AI)
+endif()
+find_package(LibArchive)
+if(NOT LibArchive_FOUND)
+  message(FATAL_ERROR "libarchive not found")
+else()
+  include_directories(SYSTEM ${LibArchive_INCLUDE_DIRS})
+  list(APPEND LIBS ${LibArchive_LIBRARIES})
+endif()
 
 # Require exiv2 >= 0.27.2 to make sure everything we need is available
 find_package(Exiv2 0.27.2 REQUIRED)