feat(cli): derive the accounts drive layout from the built machine

[brunomenezes]

Summary

Derives the accounts-drive part of the emergency withdrawal configuration from the built machine, as proposed in #516. A raw drive or nvram marked accounts_drive = true turns emergency withdrawal on. The CLI reads log2_max_num_of_accounts and accounts_drive_start_index from .cartesi/image/config.json, so developers no longer compute them, and a value that no longer matches the machine can't be deployed without an error.

The change is additive. [withdrawal.config] keeps working, every key in it becomes optional, and layout keys written by hand are checked against the build. It's an alternative to #547, which removes [withdrawal.config]. Both use the same marker syntax.

Warning

The last commit, wip(temp): build the SDK with the rollups-node next/2.0 nvram fix, is temporary and blocks the merge. The machine-tool in rollups-node v2.0.0-alpha.13 can't prove an accounts drive kept in an nvram; next/2.0 (e9da54b2) can. This commit builds the PR's SDK with that unreleased node and runs the nvram layouts against it. Once a rollups-node release ships the fix: drop the commit, unskip the nvram layouts, and bump the SDK's node version.

How it works

cartesi build
  → parse cartesi.toml        (one marked drive; raw and unmounted; power-of-two size)
  → build drives and nvrams   (a marked flash drive gets mount:false)
  → store the machine
  → find the accounts drive   (by label in .cartesi/image/config.json → start, length, device)
  → derive the config         (start index, number of accounts, devnet defaults; written keys must match)
  → print the layout and the --withdrawal-config

cartesi run
  → derive the config         (before starting the environment, when a build exists)
  → start the environment → deploy with the derived config
  → [rebuild] re-read cartesi.toml → derive → undeploy → deploy

Features

• The marker. accounts_drive = true goes on a raw flash drive (builder = "empty", format = "raw") or on an nvram. accounts_drive_size keeps the accounts at the beginning of a larger drive, for applications that keep their own state after them, like a DEX.

  [nvrams.state]
  size = "12Mi"
  accounts_drive = true
  accounts_drive_size = "4Mi"

• [withdrawal.config] and all its keys are optional.
  • log2_max_num_of_accounts and accounts_drive_start_index are derived.
  • If either is written, it must match the build. Otherwise build and run fail, showing both values and the likely cause.
  • Unknown keys are rejected.
• account_size in bytes.
  • It must be a power of two of at least 32.
  • It replaces log2_leaves_per_account, which still works but must agree with it.
  • The devnet withdrawal output builder needs 32-byte accounts, so any other size with it fails, saying what to set.
• Devnet defaults.
  • guardian defaults to devnet account 1. It's separate from the node's signer, so the two don't compete for nonces, and cartesi foreclose still finds it in the node's mnemonic.
  • withdrawal_output_builder defaults to the devnet TestUsdWithdrawalOutputBuilder.
  • On a fork, both must be set.
• cartesi build prints the layout:
  • the drive and its guest device, start, size and number of accounts;
  • each [withdrawal.config] value, marked as default, derived, or matching cartesi.toml;
  • a pasteable --withdrawal-config.
• cartesi run derives the config before every deploy.
  • It fails before starting the environment if the config can't be derived.
  • On a rebuild, it re-reads cartesi.toml and derives the config before undeploying the running application.

Fixes

• accounts_drive_start_index lost precision above 2^53. The node reads it as a uint64. It's now a bigint all the way from cartesi.toml (an integer, or a quoted decimal or hex string) to the --withdrawal-config JSON.
• A drive size of "4Mi" meant 4 bytes. Drive size and extra_size now use the nvram size parser:
  • IEC and decimal units, all as powers of 1024, up to PB;
  • sizes beyond 2^53 bytes are rejected.

Refactoring

• Drives and nvrams share one size parser, parseSize. The bytes dependency is removed.
• The derivation lives in src/withdrawal.ts. resolveWithdrawal locates the accounts drive and derives the config in one place, for both build and run. stringifyWithdrawalConfig in exec/rollups.ts writes the node's JSON.
• The fund recovery test helpers move to tests/integration/devnet.ts: devnet accounts, deposits, building, starting and stopping an application, and waiting for an accepted epoch.
• recovery.test.ts now runs on the same test application as the layout tests. The old erc20-withdrawal fixture is removed.

CI and tooling

These are temporary, in the wip(temp) commit only:

• packages/sdk/temp/fetch.sh downloads the next/2.0 build artifact, checks it against GitHub's published sha256, and extracts the two .deb files.
• The Dockerfile installs those debs, checking their pinned sha256.
• cli.yaml calls sdk.yaml, which builds and pushes ghcr.io/cartesi/*:pr-<number>. The tests then run against those images through CARTESI_TEST_SDK and CARTESI_TEST_RUNTIME_VERSION.

Verification

Full suite, unit and integration:

SDK	Pass	Skip	Fail
Released 0.12.0-alpha.44, without the temporary commit	402	13	0
The temporary commit's SDK	408	1	0

• Skips: on the released SDK, 12 of the skips are the three nvram layouts. The remaining skip is pre-existing (docker.test.ts, sqfs drive).
• Each commit on its own: passes tsc, biome ci and the unit tests.
• Unit tests: 276 → 348.

withdrawal-layouts.test.ts:

• The test app: a fresh application that finds its accounts drive by label (memoryrange, readmmap/writemmap). Its cartesi.toml only marks the drive, so it can only withdraw if the derivation is correct.
• The flow per layout: build, deploy, deposits, foreclosure, then every balance proven and withdrawn. Each balance is checked as 0 before and as the full amount after.
• Flash-drive layouts: a whole flash drive, and the first 4 MB of a 24 MB flash drive.
• nvram layouts: a whole nvram; the first 4 MiB of a 12 MiB nvram; and an nvram image holding application state after its accounts. On alpha.13 these fail in the machine-tool with accounts drive not found in stored machine. With next/2.0 they pass.
• A stale accounts_drive_start_index: fails the build.

recovery.test.ts runs on the same application. It writes all five keys by hand, working out the layout from the built machine's config.json independently of the CLI, and they pass the CLI's check against the build.

[changeset-bot]

🦋 Changeset detected

Latest commit: 740fabb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@cartesi/cli	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🟢	Lines	91.05% (🎯 0%)	12368 / 13583
🔵	Statements	91.05%	12368 / 13583
🔵	Functions	81.69%	281 / 344
🔵	Branches	0%	0 / 0

📁 File Coverage (20 files)

File	Lines	Statements	Functions	Branches	Uncovered Lines
apps/cli/src/base.ts	🔴 39.89%	🔴 39.89%	🟡 64.71%	🔴 0%	49, 70-71, 79-83, 88, 110-1...
apps/cli/src/builder/directory.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/builder/docker.ts	🟢 86.72%	🟢 86.72%	🟡 66.67%	🔴 0%	75-77, 79, 109-111, 169-178
apps/cli/src/builder/empty.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/builder/none.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/builder/nvram.ts	🟢 96.88%	🟢 96.88%	🟢 100%	🔴 0%	27
apps/cli/src/builder/tar.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/commands/foreclose.ts	🔴 49.09%	🔴 49.09%	🔴 50%	🔴 0%	69, 91-145
apps/cli/src/commands/refund.ts	🔴 38.2%	🔴 38.2%	🟡 66.67%	🔴 0%	55-109
apps/cli/src/commands/withdraw.ts	🔴 46.58%	🔴 46.58%	🟡 73.33%	🔴 0%	95-123, 277-278, 301-356, 3...
apps/cli/src/compose/anvil.ts	🟡 79.25%	🟡 79.25%	🟢 100%	🔴 0%	19-29
apps/cli/src/compose/builder.ts	🟢 99.79%	🟢 99.79%	🟢 100%	🔴 0%	228
apps/cli/src/compose/bundler.ts	🔴 4.82%	🔴 4.82%	🔴 0%	🔴 0%	8-40, 44-75, 79-92
apps/cli/src/compose/common.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/compose/database.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/compose/explorer.ts	🔴 6.67%	🔴 6.67%	🔴 0%	🔴 0%	10-38, 43-55, 59-72
apps/cli/src/compose/node.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-
apps/cli/src/compose/passkey.ts	🔴 8.33%	🔴 8.33%	🔴 0%	🔴 0%	9-18, 23-42, 46-59
apps/cli/src/compose/paymaster.ts	🔴 7.69%	🔴 7.69%	🔴 0%	🔴 0%	8-21, 25-44, 48-61
apps/cli/src/compose/proxy.ts	🟢 100%	🟢 100%	🟢 100%	🔴 0%	-

[tuler]

Thanks for this. I'm closing #547 in favor of this PR. The test work here is what makes the difference, and it's worth keeping. My main disagreement is with two premises in the config design. Most of the extra source code serves those premises, so dropping them should shrink the PR considerably.

Keeping [withdrawal.config]

We're in alpha, and this key isn't a good one to preserve. [withdrawal.config] reads as a subtable of a [withdrawal] table that doesn't otherwise exist. The compatibility also doesn't hold in practice:

• Existing configs break anyway. An existing [withdrawal.config] is rejected unless a drive is marked, so every user has to edit their cartesi.toml regardless.
• Old values fail the new check. The CLI requires cartesi-machine ^0.21.0, and 0.21.0 is where the emulator started placing drives just past RAM (machine-emulator d573f1b8). Start indexes computed for the old layout (1 << 55 plus a 1 << 52 stride) no longer match the build. The changeset says an existing [withdrawal.config] "keeps working once its drive is marked", but in practice it won't.

I'd remove it. The design proposed in #516 had [withdrawal] hold only optional overrides: guardian, withdrawal_output_builder, and the account size. When someone still writes [withdrawal.config], the CLI should fail with a message explaining how to migrate.

Overriding accounts_drive_start_index and log2_max_num_of_accounts

I couldn't find a use case for writing these by hand. Because they must equal the derived values, writing them never changes what gets deployed; they only add a check.

• Pinning the layout: any rebuild changes the template hash and needs a new deployment, which derives the values again, so there's nothing to protect.
• Accounts at an offset: that would need a different value, which the match check forbids. The node's proof tool also requires an exact start.
• Old emulator layouts: can't happen, since ^0.21.0 is required.
• An independent check in tests: this is the only current use, in recovery.test.ts. The test can compare its own calculation from config.json against the CLI's printed or derived config, without a user-facing key.

I'd reject both keys with "derived from the built machine, remove it".

This also makes the bigint work unnecessary: parseRequiredBigInt, the global integersAsBigInt: "asNeeded" TOML option, the hand-built JSON in stringifyWithdrawalConfig, and the precision changeset. The derived index is the start divided by the size, and resolveAccountsDriveLayout already rejects any start above 2^53, so the index can't exceed 2^53. A large value can only come from the user, and it then fails the match check. It was a real bug on the base branch, which passes the written value straight through, but it goes away once the value is derived.

Smaller points

• Account size. account_size and log2_leaves_per_account are two keys for one value and need a check that they agree. I'd keep only account_size, which is the friendlier one.
• Fork requiring guardian. A fork still has the devnet accounts funded, and the node still signs with the devnet mnemonic, so the default guardian works there. Only withdrawal_output_builder truly has no default on a fork.
• Changesets. There are four, all marked patch, for a feature. Consider a single minor one, and drop the claim that existing configs keep working.
• Fractional sizes. Something like accounts_drive_size = "1.1Ki" reaches BigInt() and throws a raw RangeError instead of a config error. parseSize should require an integer number of bytes.
• Drive size parser. The size fix in db093ad ("4Mi" meant 4 bytes) is a real bug fix that stands on its own. It could land as a separate PR, so it isn't held up by the node release.

What's good here

• Tests. The label-based test app, which can only withdraw if the derived layout is correct, and the 5 layouts proven end to end, including the DEX-like nvram image. This is what was missing.
• The nvram proving gap. Getting the machine-tool fix into rollups-node next/2.0, and skipping the nvram layouts until it's released.
• Guardian default. Devnet account 1 means the guardian doesn't compete for nonces with the node signer when cartesi foreclose runs.
• Account size check. Rejecting a non-32-byte account size with the devnet output builder.
• Rebuild from the run menu. Re-reading cartesi.toml, and deriving the new config before undeploying.