retroarch: fix changing the video filter in game

[qzydustin]

Summary

What does this PR do?

Changing the video filter in game with threaded video crashes RetroArch (Gameboy4x_Pocket), leaves a black screen (2xBR) or garbles the picture (ntsc_crt).

Backports three upstream fixes newer than our RetroArch:

• 0013: 8228e15, bound the staged frame by the buffer size (the crash)
• 0014: 235f684cbc, use the filter's pixel format only while filtering, and set video up again when the filter is removed
• 0015: ce362265c6, my menu: selecting a video filter sets video up again, as removing one does libretro/RetroArch#19732 (merged upstream): set video up again when a filter is selected, and on Video Filter Enable when the filter changes the pixel format

They can be dropped at the next RetroArch bump.

---

Testing & Verification

• Device(s) Tested: R36S clone (RK3326), mGBA
• Build Artifacts / Commit: 3b2437d

Test Steps

1. In a GBA game, pick Gameboy4x_Pocket, 2xBR, Scanline2x and ntsc_crt in turn (Settings > Video > Video Filter).
2. Toggle Video Filter Enable with each.
3. Remove the filter.

Test Results & Evidence

Before: Gameboy4x_Pocket crashes (exit 139), 2xBR is black, ntsc_crt is garbled.
After: all filters apply without a crash; toggling and removing give the right picture. Toggling reinitialises video only for ntsc_crt.

---

Additional Context & Reviewer Notes

0013 and 0014 are upstream with offsets adjusted; 0014's new helper checks XRGB8888 only, as this version has no 10-bit formats. 0015 is the menu part only: the filter path setting is in menu_setting.c here, and the test harness is left out.

---

AI Assistance Disclosure

While ROCKNIX has no restrictions on using AI tools, transparency helps reviewers set the proper context during code review.

Did you use AI tools (e.g., ChatGPT, Copilot, Claude) to assist with this pull request?

• No Written entirely without AI tools.
• Partially AI was used for minor assistance (e.g., refactoring, writing tests, debugging, documentation).
• Yes AI generated a major portion of the code or implementation.