Skip to content

accel/amdxdna: support graceful hw context destroy and restore on aie4 - #1635

Merged
maxzhen merged 1 commit into
amd:mainfrom
donwalkarsoham:aie4-graceful-ctx-restore
Sep 23, 2026
Merged

maxzhen merged 1 commit into
amd:mainfrom
donwalkarsoham:aie4-graceful-ctx-restore

Conversation

@donwalkarsoham

Copy link
Copy Markdown
Contributor

Suspend destroys hw contexts non-gracefully, so firmware discards their state and resume rebuilds from scratch. Firmware can instead park a context and return a restore_id for a later create to replay.

Destroy gracefully on suspend, keep the restore_id and pass it to the next create. The parked state lives in a firmware context restore pool that does not survive a firmware suspend, so copy it into a driver buffer once all contexts are destroyed and hand it back after the partition is recreated. Reset and TDR stay non-graceful, and an FLR drops both the ids and the pool. If firmware returns restore_id 0, or the pool cannot be saved, the ids are cleared and parked jobs aborted so resume builds fresh contexts.

@xdavidz

xdavidz commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Suspend destroys hw contexts non-gracefully, so firmware discards their state and resume rebuilds from scratch. Firmware can instead park a context and return a restore_id for a later create to replay.

Destroy gracefully on suspend, keep the restore_id and pass it to the next create. The parked state lives in a firmware context restore pool that does not survive a firmware suspend, so copy it into a driver buffer once all contexts are destroyed and hand it back after the partition is recreated. Reset and TDR stay non-graceful, and an FLR drops both the ids and the pool. If firmware returns restore_id 0, or the pool cannot be saved, the ids are cleared and parked jobs aborted so resume builds fresh contexts.

Will you be able to develope a shim_test for this too?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds AIE4-specific support for preserving and restoring hardware context state across runtime auto-suspend/resume by using firmware “graceful destroy” restore IDs and persisting the firmware restore-pool across the suspend power boundary.

Changes:

  • Implement graceful hwctx destroy on suspend, capturing per-context restore_id and saving the firmware context restore pool into a driver buffer for resume.
  • Restore the saved restore pool on resume before recreating contexts, and fall back to fresh context creation if pool save/restore fails (clearing restore IDs and aborting parked jobs).
  • Add a shim_test that holds an idle context across multiple runtime auto-suspend cycles and submits work after each resume to guard the restore path from regressions.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
test/shim_test/shim_test.cpp Adds an AIE4 + native-driver device filter and registers the new suspend/restore regression test.
test/shim_test/io_test.cpp Implements runtime PM polling via sysfs and adds TEST_ctx_restore_across_suspend.
drivers/accel/amdxdna/amdxdna_ctx.c Resumes the device before hwctx erase/destroy so firmware restore data can be reclaimed when destroying a suspended context.
drivers/accel/amdxdna/aie4_pci.h Adds restore_id to hwctx private state and adds a device-level saved restore-pool handle plus message prototypes.
drivers/accel/amdxdna/aie4_pci.c Saves/clears/restores the firmware ctx restore pool across suspend/resume; switches suspend path to graceful destroy for non-clean shutdowns.
drivers/accel/amdxdna/aie4_msg_priv.h Introduces new AIE4 mailbox opcodes and request/response structs for restore-pool get/set.
drivers/accel/amdxdna/aie4_message.c Implements get-size/get/set message helpers for the context restore pool.
drivers/accel/amdxdna/aie4_ctx.c Threads restore IDs through graceful destroy and into subsequent create; aborts parked jobs when graceful destroy returns restore_id=0.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@maxzhen
maxzhen requested a review from wendyliang25 August 28, 2026 20:10
Comment thread drivers/accel/amdxdna/aie4_ctx.c Outdated
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch from d240b03 to 401a100 Compare August 28, 2026 23:43
@donwalkarsoham

Copy link
Copy Markdown
Contributor Author

Suspend destroys hw contexts non-gracefully, so firmware discards their state and resume rebuilds from scratch. Firmware can instead park a context and return a restore_id for a later create to replay.
Destroy gracefully on suspend, keep the restore_id and pass it to the next create. The parked state lives in a firmware context restore pool that does not survive a firmware suspend, so copy it into a driver buffer once all contexts are destroyed and hand it back after the partition is recreated. Reset and TDR stay non-graceful, and an FLR drops both the ids and the pool. If firmware returns restore_id 0, or the pool cannot be saved, the ids are cleared and parked jobs aborted so resume builds fresh contexts.

Will you be able to develope a shim_test for this too?

Yes, a shim test is added in the PR.

@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch 2 times, most recently from 39213e1 to d9adf1b Compare September 2, 2026 01:31
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch 2 times, most recently from 2fdea67 to d45ffb9 Compare September 3, 2026 23:49
@xdavidz xdavidz added documentation Improvements or additions to documentation do-not-merge labels Sep 4, 2026
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch from d45ffb9 to 2fa30a0 Compare September 4, 2026 00:48
Comment thread drivers/accel/amdxdna/aie4_ctx.c
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch 2 times, most recently from 1033f78 to 6283c7d Compare September 9, 2026 23:56
Comment thread drivers/accel/amdxdna/aie.c Outdated
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch 4 times, most recently from 675f949 to bd584a4 Compare September 11, 2026 18:44
Comment thread drivers/accel/amdxdna/aie4_ctx.c Outdated
@xdavidz xdavidz removed documentation Improvements or additions to documentation do-not-merge labels Sep 11, 2026
Comment thread drivers/accel/amdxdna/aie4_ctx.c Outdated
@donwalkarsoham

Copy link
Copy Markdown
Contributor Author

Retest this please

@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch from e23643b to 87e4f90 Compare September 17, 2026 18:35
xdavidz
xdavidz previously approved these changes Sep 17, 2026

@xdavidz xdavidz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comments look good to me, thanks for handling this.
Please double check with @wendyliang25 if this handling look ok by her from TDR point of view.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate findings affect parked-job recovery, DMA cache coherency, and suspended cleanup.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

drivers/accel/amdxdna/aie4_pci.c:472

  • After GET_CTX_RESTORE_POOL firmware writes this DMA buffer, but this path never invalidates the CPU cache before retaining it for the later SET. On the non-coherent dma_alloc_noncoherent path, the resume can therefore hand firmware stale pool bytes; the existing telemetry read path flushes after a device write for the same reason. Flush the buffer after a successful get.
	ret = aie4_msg_get_ctx_restore_pool(ndev, to_dma_addr(buf_hdl, 0),
					    to_buf_size(buf_hdl));
  • Files reviewed: 8/8 changed files
  • Comments generated: 3
  • Review effort level: Lite

Comment thread drivers/accel/amdxdna/aie4_pci.c
Comment thread drivers/accel/amdxdna/amdxdna_ctx.c
Comment thread test/shim_test/io_test.cpp
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch 3 times, most recently from 1629cef to 4b46684 Compare September 18, 2026 22:42
@donwalkarsoham

Copy link
Copy Markdown
Contributor Author

🟡 Changes recommended

Unresolved critical and moderate findings affect parked-job recovery, DMA cache coherency, and suspended cleanup.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

drivers/accel/amdxdna/aie4_pci.c:472

  • After GET_CTX_RESTORE_POOL firmware writes this DMA buffer, but this path never invalidates the CPU cache before retaining it for the later SET. On the non-coherent dma_alloc_noncoherent path, the resume can therefore hand firmware stale pool bytes; the existing telemetry read path flushes after a device write for the same reason. Flush the buffer after a successful get.
	ret = aie4_msg_get_ctx_restore_pool(ndev, to_dma_addr(buf_hdl, 0),
					    to_buf_size(buf_hdl));
  • Files reviewed: 8/8 changed files
  • Comments generated: 3
  • Review effort level: Lite

INVALID. drm_clflush_virt_range() at line 469 already writes back and invalidates, and the CPU never touches the buffer between the GET and the SET. Its own cited precedent flushes after only because the CPU is about to read.

wendyliang25
wendyliang25 previously approved these changes Sep 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

User-mode abort handling and bidirectional DMA mapping must be corrected.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (3)

Comment thread drivers/accel/amdxdna/aie4_ctx.c
Comment thread drivers/accel/amdxdna/aie4_pci.c
Suspend destroys hw contexts non-gracefully, so firmware discards their
state and resume rebuilds from scratch. Destroy gracefully instead and
pass the returned restore_id to the next create. That state lives in a
firmware restore pool that a firmware suspend wipes, so save it into a
driver buffer and hand it back before the contexts are recreated. Reset
and TDR stay non-graceful, and an FLR drops the ids and the pool.

A graceful destroy that returns no id preserved nothing, so the suspend
aborts that context's parked jobs once it is gone. Firmware reports that
as NO_RESTORE.

Add shim tests for a context held across auto-suspend, several held so the
pool has more than one entry, and one destroyed while suspended.

Signed-off-by: Soham Donwalkar <Soham.Donwalkar@amd.com>
@donwalkarsoham
donwalkarsoham force-pushed the aie4-graceful-ctx-restore branch from 1629cef to ac4e73e Compare September 22, 2026 23:09
@xdavidz
xdavidz requested a balanced review from Copilot September 23, 2026 21:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Firmware state persistence, runtime power management, and context teardown interactions require final hardware-backed human validation.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@maxzhen
maxzhen merged commit efddccd into amd:main Sep 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants