Conversation
The AIE4 route has its own decode loop and never touched the DECODING_TIME profiler, so every run reported "Decoding time: 0 us" and an "-nan(ind)" decode speed, and meta_info.decoding_duration stayed zero on all four generation endpoints. Record the profiler around each forward, exactly as the shared decode loop does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drives the full hardware matrix from one script: pinned-package check, the CLI semantic and repeated-load cycles through a PTY, both REST APIs in streaming and non-streaming form, cancellation and recovery, the 4095/4096 boundary, and the descriptive performance record. JSON request bodies are written to files and read back by curl with "@file". Passing a body inline lets PowerShell's native-argument quoting strip the double quotes, which is why the earlier cancellation and boundary probes recorded a server-side JSON parse error instead of a result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A header value containing a space is split into two native arguments, so curl took the second half as another URL and reported "Could not resolve host: application" while sending the request without the intended content type. The colon form carries the same meaning and cannot split. Also give every REST call a timeout and fail loudly when a cancelled stream does not end, so a wedged request reports instead of hanging the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chizamd
force-pushed
the
feature/phi4-gguf-aie4
branch
from
September 12, 2026 09:20
08277ef to
95c5c82
Compare
chizamd
marked this pull request as ready for review
September 14, 2026 23:26
chizamd
force-pushed
the
feature/phi4-gguf-aie4
branch
from
September 16, 2026 20:05
5673859 to
b471e12
Compare
alfxu-amd
force-pushed
the
feature/phi4-gguf-aie4
branch
from
September 17, 2026 17:28
b471e12 to
8a2661f
Compare
Records the acceptance run on the AIE4 development machine at 8772108: load to serving 44.2/47.4/49.1 s, cold TTFT 4.21 s, warm TTFT 65.0 ms, decode 21.3 tok/s over REST and 35.8 tok/s in a warm CLI session, with the machine, power scheme, commits, revisions and file hashes it came from. Performance is descriptive; no threshold is claimed. Two numbers are deliberately not presented as throughput. The record's load_duration is 1.5 us, which is the Ollama-compatible field on an already warm server rather than a model load, so load was measured separately over three fresh processes. The ten fresh-process CLI cycles report 3.70-20.26 tok/s because each pays the one-time kernel and ELF setup inside its own eight-token window. The model card separates the two context limits that were previously conflated. The 4096 ceiling is a correctness boundary: it is Phi-4-mini's rope.scaling.original_context_length, LongRoPE selects factors by sequence length, and only the short branch is derived here. The further step down to 4095 is this frontend's own conservatism so an admitted request can always finish, and is not imposed by corelib. Also records the one unexplained reply that degenerated into a markdown image URL, and corrects the emitter comment: the unbounded ConvertTo-Json allocation reproduces on PowerShell 7.0.0 as well, so a newer engine is not a fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Model load is the one phase no profiler covered, and on this backend it is the largest single cost a user waits through. The runner now reports it for every backend, and the AIE4 engine breaks its own load into shape planning, GGUF resolution, host preparation, weight requantization and device allocation, printed when FLM_AIE4_PROFILE_LOAD is set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The helper was local to the runner, so `flm serve` -- the path that actually waits 45 s on this backend -- reported nothing. Move it to debug_utils and call it from ensure_model_loaded, which is the single funnel for the server's initial load and every later model switch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Engine load is only 16 s of it. The startup SHA-256 check over the 4 GB GGUF is ~28 s, 62% of the wait, and happens before the engine is constructed. Weight requantization is 15 s. Everything else is under a quarter of a second. The integrity check is also ~8x slower than the work needs: Get-FileHash over the same file on the same machine takes 3.67 s against ~28 s for calculate_file_sha256, which uses a portable pure-C++ SHA-256 with no hardware acceleration. That cost is paid by every model on every startup and pull, not just this one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ead hint Two independent costs in the ~45 s startup, measured not assumed. The larger one is the startup integrity check: is_model_downloaded re-hashed every pinned file on every launch, ~28 s of SHA-256 over the 4 GB GGUF, 62% of the wait. That is a pull-time concern; the parameter to skip it already existed and simply was not passed. The run and serve paths now ask for status only. `flm pull` and `flm check` are unchanged and still verify in full, so a corrupt file is still caught -- at the next explicit check rather than at every launch. The smaller one is the packer. corelib treats a threads hint of 0 as ONE, deliberately, and this requantizing path is compute-bound and scales with it. The hint is per-create and the creates stay serialized. corelib's header records that loading a model with 8 CONCURRENT creates on this entry point failed 2 of 10 with all-zero output -- token id 0 at every step -- against 0 of 10 serialized and 0 of 10 for the ONNX path, with attribution open and the leading hypothesis an incomplete host-to-device sync. That is the faster configuration and it is deliberately not taken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Re-measured after removing the per-launch re-hash and giving the packer a thread hint: process launch to serving 5.1/5.3 s against 44.9/46.3 s, with requantization at 2.5/3.0 s against 15.3/14.9 s. Output re-verified after the packing change; no degeneration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A model family does not necessarily have an engine for both NPU generations, so `flm list` and tag resolution have to be pruned to the generation this binary can actually run. model_list.json grows an optional platform tag per entry and model_list prunes itself to it; an entry that says nothing is aie2p, which is every entry that existed before. Which generation that is comes from FLM_ENABLE_AIE4, the same option that decides whether ryzenai-corelib or FastFlowLM's own kernels are compiled in. The two share no engine and no binary carries both, so the platform is a build-time constant (`utils::build_npu_platform()`) rather than something to go and ask the hardware about.
include/models/phi4 held the FLM NPU headers and the corelib headers side by side, so an #include said nothing about which engine it belonged to. Mirror the layout common/models/phi4 already uses: include/models/phi4/flm/ phi4_npu.hpp, phi4_npu_sequence.hpp include/models/phi4/corelib/ phi4_corelib_*.hpp A grep for models/<model>/corelib/ now finds everything the corelib path pulls in. Moves only — no content change beyond the include paths in builtin_backends.cpp, automodel.hpp, the corelib sources and the tests, plus the layout section and links in common/models/README.md. Nothing globs or lists include/models, so no CMake change was needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ckend The backend opened the model directory itself to cross-validate its package, and the frontend opened the same file again for the chat template, so a load read tokenizer_config.json twice and the file-access audit went red. Pass it down instead. load_model reads it once and hands the parse to both _shared_load_backend and setup_tokenizer, so the directory layout stays the frontend's knowledge: a backend is a pluggable lower layer and should be given what it needs rather than knowing which file to open. That also drops a filesystem dependency from the backend, and it scales -- a second backend needing the same file would otherwise have copied the read a third time. Cross-validation is unaffected. ValidatePhi4Contract draws its independence from comparing two different sources, the GGUF metadata against the shipped config, not from which layer called open. The test stub mirrors the real backend by construction, so it takes the parse from the context too. Also stop the test harness exiting on the first failure. RunTest called exit(1), so a suite with four broken tests reported one and each fix merely uncovered the next -- which is how three of these stayed hidden. It now counts failures, reports them all, and forces a nonzero exit from atexit because every main here ends with an unconditional PASS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_shared_generate had three exits that never recorded a stop reason. The one that matters is the decode cap: the loop tested it in the `while` condition, so running out of window left `reason` at its EOT_DETECTED default and the caller was told the model had chosen to stop. That is what becomes `done_reason` over the API, so a truncated reply was indistinguishable from a complete one. The cap is now an explicit break that records MAX_LENGTH_REACHED, and the two early returns -- EOS, and already at the cap on entry -- assign the reason they had computed but were dropping. A poisoned model also described itself two ways: the request that broke it got "unload/reload is required", every later one got "the model must be reloaded" from the shared guard. Same condition, same words now, keeping the backend id the guard usefully added. The cancellation test asserted a poll count, which only held for the old loop shape: it cancelled after a fixed number of predicate calls, so a loop that consults the predicate once per iteration instead of twice "failed" it while behaving correctly. It now cancels once a decode step has actually run and still requires that no further forward is dispatched, which is the property worth holding and does not depend on polling cadence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Weight requantization is the whole of model load on this backend. The creates are independent -- each reads its own mapped range of the GGUF and produces its own object -- so they now run across a pool of 8 rather than in sequence. The per-create thread hint goes back to corelib's default of one. The parallelism is real threads we own, which also means it is not subject to whatever thread limits the surrounding environment imposes on the packer; a session that had the packer pinned to one thread was paying 30 s where the same binary took 3 s elsewhere. Order of completion is no longer defined, so the tests that indexed the create log positionally now look creates up by the mapped address they packed from. That is the stronger assertion anyway: the lookup requires exactly one match, so it proves each weight was packed from its own range exactly once, which positional indexing never did. The fake corelib held its state mutex for the whole of every entry point, so creates serialised inside it however many threads called in -- the concurrency could not be observed and would not have been exercised. It now steps outside the lock for a short packing window, which is what a real create does, and the engine test requires the observed peak to be above one and within the budget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The load figures now come from a different machine and a different packer than the generation figures, so the page says which is which rather than presenting one set. Machine A keeps the acceptance run and everything derived from it, at the commit it was measured on, and is marked as predating the restructure and the concurrent packer. Machine B carries load only, and says plainly that TTFT, decode and the acceptance matrix were not re-run there. Machine B's load is reported as a range. It varies 3.50-13.51 s across six runs on an idle machine, so a single figure would misrepresent it; the consistent ~30 s it replaced is quoted alongside because that contrast is the result, not the best sample. Also drops the description of corelib's concurrency observation. Repeating an open attribution here amplified it beyond what the evidence supports, and the statement that creates stay serialized is no longer true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It now describes the concurrent packer the code actually uses, and drops the repetition of corelib's open attribution along with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Requantizing the 161 weights from Q8_0 is effectively the whole of model load. corelib can hand the packed bytes back and load them again later -- mapping the file rather than copying it -- so the refit need only happen the first time a given GGUF is loaded. Three entry points are added to the API table for it: weights_copy_data and the two ..._weights_create_from_file. All 161 blobs go in one file with a JSON index beside it, written last and renamed into place so a data file without a matching index is never used. Both directions walk the same slot order, so an index entry always refers to the weight it was written from; the creates complete out of order but are assigned by slot, not by completion. The cache is keyed on the GGUF's size and write time, the corelib version, the group size and the weight count. Not on the GGUF's content: hashing 4 GB costs more than the packing the cache exists to avoid, which would defeat it. That trade is safe to make here because corelib independently rejects a slice whose length is not exactly what the descriptor packs to, so a stale-but-plausible cache cannot quietly become the wrong weights. A miss, a stale key, an unreadable index or a failed bind all fall back to packing; a partial bind releases everything rather than leaving half a model. FLM_AIE4_WEIGHT_CACHE redirects the cache or, set to 0, disables it. The engine tests disable it: a cache written by one test would otherwise make every later one load instead of pack, and the packing assertions would quietly stop testing anything. Whether this is faster than packing is a measurement, not yet a claim -- corelib's own guidance is that caching only pays for a large blob. The load profile reports which path ran and what writing the cache cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cache whose key no longer matches was ignored and then eventually overwritten by the next successful write. Two gigabytes stayed on disk in between, and stayed indefinitely if that write never happened -- a failed pack, a read-only directory, a machine that never loads the model again. It is now deleted at the point it is known to be useless, before repacking, together with the temporaries an interrupted write leaves behind. The load profile says how much was reclaimed. Failing to delete is not a reason to fail a load, so this is best effort like the rest of the cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backend id was a mix of implementation and hardware: `flm_npu` named the code, `corelib_aie4_gguf` named the code, the vendor library and the file format. Neither said the one thing a caller needs, which is the NPU generation. A family has at most one engine per generation, so the generation is the only axis of choice there ever was. So the backend id is now the platform id: `aie2p` and `aie4`, the same strings `model_list.json` already uses for `supported_platforms`, and the same string `build_npu_platform()` returns. `resolve_backend_id` no longer translates between two vocabularies. The tree follows the names: `common/corelib/` and `include/corelib/` become `aie4/`, `models/phi4/flm/` becomes `phi4/aie2p/`, `models/phi4/corelib/phi4_corelib_*` becomes `phi4/aie4/phi4_aie4_*`, and the test suite `phi4_corelib_aie4/` becomes `phi4_aie4/`. Class `phi4_corelib_aie4` is `phi4_aie4`, and `CorelibProcessGuard` is `Aie4ProcessGuard`. The build option `FLM_ENABLE_CORELIB_AIE4` becomes `FLM_ENABLE_AIE4` for the same reason: it selects a generation, not a library. It is also now the only thing that selects one, so its name should say so. `acquire_npu_device()` becomes `open_npu_device()`, which is what it does. No behaviour changes. The files under `aie4/` that wrap corelib's C API keep their `corelib_` names, because there the vendor library is the subject. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
alfxu-amd
force-pushed
the
feature/phi4-gguf-aie4
branch
from
September 17, 2026 17:43
8a2661f to
b5a05ac
Compare
chizamd
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Withdrawn. Description removed.