Dockerize backend - #11
Merged
Merged
Conversation
Design for turning this repo into a starter template someone can clone, point at their own source database, and edit: two run modes selected by one line of configuration, four Example Source Databases, and a development loop that does not require rebuilding an image. Records the rejected Compose layouts with the specific failure each has, and the two load-bearing mechanisms verified against Compose v2.38.2 rather than assumed — overlay volume replacement keyed on mount target, and the Compose file list being read from the environment file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backend image booted, logged that it was running, and then answered every request with 500 "openapi.validator: spec could not be read at /openapi.yaml". The contract it validates against lived at the repo root, outside the backend's Docker build context, so it was never copied into the image — and the validator loads it lazily on first request, so nothing failed at startup. A container that looked healthy and served nothing. Move the contract beside app.ts, inside the build context. Both packages still generate their types from that one file, so the API stays single-sourced. Two further defects surfaced while fixing it: The backend had no .dockerignore, so COPY / ./ copied the host's node_modules over the ones installed in the image — macOS-native binaries into a Linux container — along with local env files and logs. The root route was unreachable. It is registered after the OpenAPI validator, which rejects any path absent from the spec, so it 404'd even on a machine where the spec resolved. It now has an explicit exemption alongside the auth routes, which also gives the backend a liveness probe for Compose to depend on. Adds the first test at the existing HTTP seam, covering assembly rather than any one route: if the contract cannot be resolved, every test here fails at once. Verified against a built image — root route 200, token and JWKS endpoints responding, a contract-violating request rejected with 400 rather than falling through to the auth gate, and the image's node_modules the Linux ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migrate from legacy bucket definitions to the edition 3 streams format, and rename the sync file to match. The service configuration now references it under sync_config; the deprecated sync_rules key is gone, since specifying both is a hard error. The legacy format still works and is not deprecated, so this is not forced. The reason to move is that whatever ships here is what an adopter copies when writing rules for their own schema, and upstream documentation now describes only the new shape. auto_subscribe is set explicitly on the stream. It defaults to false, and the legacy format had no equivalent concept, so omitting it syncs nothing while reporting no error anywhere. Verified end to end: the service accepted the config and replicated both tables, a sync stream opened with no explicit subscription reported the stream as default and delivered all seeded rows, and a write through the API came back through sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standards axis: The test-file comment claimed the validator resolves the contract at construction. It does not — it resolves lazily on the first request, which is precisely why the container booted cleanly and only then failed everything. Corrected, and the claim that both tests are a regression net is now stated as verified rather than asserted: spec resolution runs before the ignorePaths check, so removing the contract fails both tests, including the exempted root route. Confirmed by deleting the file and watching them go red. The ignorePaths regex packed two unrelated exemptions into one opaque literal with no explanation; commented. .dockerignore used *.test.ts, which in Docker's matching does not cross a path separator, so any future src/**/*.test.ts would still have shipped. backend/README.md coined "flavour" as a domain term while the glossary is deliberately untouched; reworded. Test prose used bare "batch", which the glossary lists under Transaction's avoid-list. Spec axis: The README project layout still listed sync_rules.yaml, renamed in the previous commit — and had been stale for longer than that, still naming the backend and frontend by directory names they no longer use. self-host-plan.md described the service.yaml rename as part of the streams migration when it belongs with the directory restructure, and claimed .scratch issues were out of scope when the work is in fact tracked there. Two review findings were checked and rejected. The root route is not the only path outside the contract — only /api/data is declared, so the auth routes have always been exempt too, and a liveness probe follows that precedent. The sync config's editor schema URL resolves and is the path upstream ships on its own edition 3 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One command from a fresh clone now brings up a complete working system with no configuration and no credentials: seeded Postgres, bucket storage, the write API built from local source, and the demo client. Splits compose into a base definition plus overlays. The base is everything both modes share and is Adopter Mode on its own; examples/postgres/compose.yaml adds the source database, the demo client, and its own PowerSync config. Mode selection is the COMPOSE_FILE line in .env, so the command stays a plain docker compose up and every other subcommand behaves normally. Two pieces of wiring change now that the backend is containerised. PowerSync reaches it by service name rather than through the host, and gates startup on its healthcheck — which works only because ticket 01 made the root route reachable, so that exemption is now load-bearing for startup ordering. The signing keypair is committed as a public throwaway. Without it the backend mints a new key on every restart and PowerSync rejects tokens it just accepted with PSYNC_S2101; that failure was hit for real while verifying the previous commit. Tested directly here: after restarting the backend, both a token issued before the restart and a freshly issued one are accepted, with no PowerSync restart. config/service.yaml and config/sync-config.yaml are pulled forward from the next ticket. The base mounts ./config, and moving the demo config into examples/ left that path empty, so Compose would have created an empty directory and PowerSync would have died looking for /config/service.yaml. Verified from a down -v: all services healthy, seeded rows delivered through a sync stream with no explicit subscription, and a write round-tripping through Postgres and back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adopter Mode now works against a database this project does not own. Verified against a standalone Postgres started outside this Compose file, with its own schema and credentials, reached over host.docker.internal: the adopter's rows synced, a write through the API landed in their database, and it came back. The fixed rule was checked rather than asserted — after a full run against that external database, it contained zero powersync schemas. Bucket storage stayed in the container this project owns. Refusing to start is now a readable failure. It already failed before listening, but by throwing a raw stack trace, which reads as a bug in the adopter's code rather than a setting they have not filled in. A ConfigurationError distinguishes misconfiguration from a genuine fault, and the message names the fix, including the alternative of selecting a bundled example. An unsupported DATABASE_TYPE now lists what is supported. Written test-first at a process-level seam. The spec claimed Seam 1 covered this; it does not, and cannot — Seam 1 drives HTTP against an assembled application, and the behaviour here is a refusal to reach the point where there is anything to send a request to. The check spawns the backend and asserts on exit code and output, in the backend's existing runner rather than a third architectural seam. config/sync-config.yaml ships empty because only the adopter knows their schema, and PowerSync treats that as fatal and restart-loops on 'streams' are required. Shipping a stream that references tables which may not exist would be worse, so the template and README both state that this is expected and must be filled in before starting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Guards the two mechanisms holding the mode switch together, both of which fail silently. If an example's config mount appended to the base's instead of replacing it, or a mode selection failed to resolve, the stack would come up perfectly healthy pointing at the wrong sync rules — nothing crashes, nothing logs, and the only symptom is data that does not sync. Compose is asked to resolve each mode rather than run it, so no containers start and no images are pulled; the suite takes about a second. It asserts the service set, the project name, that exactly one directory is mounted at /config and it is the right one, and that the write API and replication point at the same database. Two assertions go past the ticket's criteria deliberately: bucket storage must point at the owned mongo service in every mode, which is the spec's one fixed rule, and PowerSync must reach the backend by service name rather than through the host. Mutation-tested: breaking the overlay's mount target turns exactly one test red and leaves the rest green. Lives at the repo root with its own runner, because the topology is a repo-level concern and the backend's suite must stay runnable without Docker. The root had no package.json and no node_modules ignore rule; both added, the latter scoped so it cannot match the lockfiles the Dockerfiles install from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding a config at the repo root silently broke the backend's suite. Vitest walks up looking for a config file; the backend had none of its own, so it inherited the root's include pattern, found nothing matching, and exited 1 reporting "No test files found" — with both of its test files sitting right there. The backend now has an explicit config. Both suites verified by running them: 4 tests in the backend, 12 at the root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds no database container at all. PowerSync already runs a Mongo replica set for bucket storage, so replication points at a second database on that same server — powersync_demo_source alongside powersync_bucket_storage. Separate databases, one process, so trying this flavour costs nothing extra. No seed script either: collections are created implicitly by the first write, so the app starts empty and fills as you use it. The sync rules project _id onto id, which is what makes documents arrive with the identifier the rest of the system keys on. The comment explaining it was wrong — inherited from upstream's wording, it claimed * excludes underscore-prefixed fields. The synced documents carry _id alongside id, so the comment now describes what actually happens; the extra field is harmless. The topology suite covers this mode too, including that it introduces no source service. Eighteen assertions across three modes. Verified: a write returned success, landed in the source database with the client-generated string as _id, and came back through sync with a usable id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Append docker-compose.dev.yaml to any mode and your working tree is mounted into the container with the watch process running: an edit is serving in about two seconds, no image rebuild. Verified in both modes by editing a route handler while the stack ran, including in Adopter Mode against an external Postgres. The keypair dependency was verified rather than assumed — a token issued before an edit triggered a reload was still accepted afterwards. That is precisely why the previous work committed a fixed pair; ephemeral keys would have fired this on every save. An anonymous volume preserves the image's node_modules. Without it the bind mount shadows them with the host's, which on macOS means binaries the container cannot run. The demo client stays out of the overlay deliberately. The overlay applies to both modes and Adopter Mode has no frontend service, so an override would try to create one with no build context. Its loop is the ordinary `pnpm dev` on the host, which reads .env.local at runtime — documented instead. pnpm generate-keys prints a replacement keypair in the form .env expects. Also documents a trap hit while testing: each mode is its own Compose project, so `docker compose down` only stops the mode currently selected in .env. Switching without bringing the old one down leaves containers holding ports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A seeded MySQL Example Source Database, selected by the same one-line switch. The binlog configuration is mounted into the server's config directory rather than applied by a setup script, because PowerSync reads the binary log and it has to be in the right shape before the server starts — by the time a script could run it is too late. That is the structural difference from the Postgres example, where wal_level is a command-line flag and the publication is created afterwards in SQL. Two users deliberately: the write API connects as root while PowerSync connects as a restricted powersync user holding REPLICATION SLAVE to read the binary log and SELECT to take the initial snapshot, showing what least privilege replication actually needs. Verified by querying the running server for the settings rather than assuming the config was read — log_bin, gtid_mode, enforce_gtid_consistency, binlog_format ROW, binlog_row_image FULL and a server id all confirmed. Seeded rows synced, and a write round-tripped through MySQL and back. The connector is Beta, stated plainly in the example README and marked in the root README's mode table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The heaviest of the four. SQL Server replicates through Change Data Capture rather than log streaming, and has no entrypoint directory to drop SQL into, so the bootstrap is a one-shot container gated on service_completed_successfully: CDC at database level, a _powersync_checkpoints table with CDC enabled on it, CDC per replicated table, and a restricted replication user. SQL Server Agent is enabled because CDC capture runs as Agent jobs — without it CDC appears enabled and captures nothing. Fixes a permission bug found by running it rather than reading about it. With CDC fully configured, replication still failed with "The user does not have permission to perform this action", preceded by VIEW SERVER PERFORMANCE STATE being denied in master. That is a server-level permission on the login; the script had granted only the database-level VIEW DATABASE PERFORMANCE STATE. Both are needed, and the two read alike in the documentation. Microsoft publishes no arm64 image, so this runs under linux/amd64 emulation with a long healthcheck start period. The other three flavours run natively. The README leads with the sharp edge rather than burying it: CDC captures a table's shape when enabled, so schema changes are not adopted automatically and an added column never arrives with nothing erroring — which matters most in a repo whose premise is replacing the schema. Verified from a dropped volume to prove the bootstrap works from nothing: CDC state confirmed on the server, seeded rows synced, and a write round-tripped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serious one: Adopter Mode's readable configuration failure was a restart loop. The backend carried restart: unless-stopped, so exit(1) respawned forever — eight times in twenty-five seconds — scrolling the message past repeatedly, and powersync gates on backend: service_healthy, so an unconfigured Adopter Mode hung on a dependency that could never pass. This was verified on the host with tsx and never in the container, which is how it survived. Now restart: on-failure:3: it stops after four attempts with the message as the last thing on screen, while a transient crash still recovers. The boot test could not have caught it, and had a matching flaw: it resolved only on 'close', so a regression that DID start the server left the promise pending until the suite timed out and leaked a listening process. It now kills the child on a deadline and asserts explicitly that it was not still running. Its stack trace assertion also hardcoded a function name, so renaming that function would have made it vacuously true; it now matches the shape of a stack frame. Spawns the local tsx rather than npx, which can reach the network on a cold cache. Four copies had already drifted apart: examples/postgres/powersync/service.yaml came through as a rename and never picked up the schema header or storage comment the other three carry, and the mysql and mssql overlays silently dropped the comment explaining why the client's URLs are localhost. The SQL Server password was written literally four times, including in the bootstrap entrypoint, while the healthcheck two blocks above correctly used the env var. Defined once as an anchor now. Database credentials across the examples are marked as public throwaways the way the signing keys already were. Topology test table had optional fields that every mode sets and every assertion dereferences, so omitting one would have passed against undefined; now required. Resolutions are memoised — the suite was shelling out to Compose once per assertion. A doc comment described a null that cannot occur. Adds the per-flavour smoke procedure the spec promised and I never wrote, as the manual check the topology suite deliberately does not replace. The Postgres one was run verbatim rather than written from memory. The README's "no configuration and no credentials" was untrue — the committed .env is both; it now says there is nothing for you to configure. The layout block showed only one of the four examples. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four documents still described a workflow that no longer exists.
backend/README.md: everything from "Running the app" onward predated Compose —
cp .env.template && pnpm start on the host, and a whole ngrok section tunnelling
to the hosted PowerSync dashboard, which has nothing to do with self-hosting. It
also cited port 8000. Replaced with how the backend actually runs, the variables
it reads and where they come from, the boot-time configuration failure, the
verifier seam, and pnpm generate-keys. The endpoint and error-classification
sections were already accurate and are kept.
diagram.png: deleted. It showed the pre-batching, pre-auth, pre-container
architecture — a request body of { crud: CrudEntry_API[] } where the contract now
takes a transaction batch, no auth at all though a Write Token and Verifier both
exist, Postgres as the only flavour, and the seed schema at a path it no longer
occupies. It was the first thing in the root README, so it was also the first
thing to mislead.
frontend/README.md: three sentences, two of them wrong. It claimed changes to
Postgres sync to clients, true for one of four flavours. Now says plainly that
the client is a test fixture bound to the demo schema and absent from Adopter
Mode, and documents the dev loop, the batching variables and type generation.
backend/.env.template listed variables as though the backend were configured
there; in Compose they come from the root .env and the overlays. It now says so,
and warns about the port 6060 collision with the container.
Also drops file line numbers from self-host-plan.md and fixes frontend's
.env.template, which told you to copy a file that has never existed under that
name. Batching defaults in the new docs were checked against DemoConnector rather
than copied from the old text.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The root .env is the file an adopter edits, and it offered `# DATABASE_TYPE=postgres` with no indication that three other values exist. The options were only written down in backend/README.md and backend/.env.template, neither of which someone configuring Adopter Mode necessarily opens. Also records the spelling trap: PowerSync's replication config in config/service.yaml calls the Postgres connector `postgresql`, while this variable is read by the write API and spells it `postgres`. Two adjacent config files, same concept, different spelling. The failure is loud rather than silent — the backend refuses to start and lists what it supports — but the note saves the trip. Adds a connection-string shape per type, and the reminder that a database on the host is reachable at host.docker.internal rather than localhost. All four shapes are the ones the working examples actually use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adopter Mode told you to set DATABASE_URI and fill in a sync config, but never that your database needs its change feed turned on first — no publication for Postgres, no binlog settings for MySQL, no CDC for SQL Server, no replica set for MongoDB. Skipping any of them produces an empty app rather than an error, which is the worst way to learn. The per-flavour prerequisites were documented, but only inside examples/, which the same README invites you to delete. Deleting it as intended removed the only place stating that a Postgres source without a publication replicates nothing. The Adopter Mode section now carries a table of what must be true of your database for each flavour, and says plainly that it is the version which survives deleting examples/. The example READMEs keep the worked SQL and the managed-hosting wrinkles. MongoDB also gained the "pointing at your own instance instead" section the other three already had: replica set requirements, post-images and privileges, the note that Azure DocumentDB shares the connector but does not support post-images, and that bucket storage stays in our container even though the bundled example shares one Mongo process between source and storage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Chriztiaan
force-pushed
the
feat/dockerize
branch
from
September 17, 2026 12:42
4ebc519 to
9baa334
Compare
ckritzinger
approved these changes
Sep 17, 2026
Chriztiaan
marked this pull request as ready for review
September 17, 2026 14:17
kobiebotha
previously approved these changes
Sep 17, 2026
Chriztiaan
dismissed
kobiebotha’s stale review
September 17, 2026 14:20
The base branch was changed.
kobiebotha
approved these changes
Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Self-hosting: one
docker compose upfrom a clean cloneTurns the demo into a starter template someone can actually take. Before this, running it meant
three terminals, two
.envfiles to fill in, and a keypair to generate first. Now a fresh clonecomes up complete — seeded database, PowerSync service, write API, demo client — with nothing to
configure, and pointing it at your own database is one line in
.env.Branches off and merges
feat/base(PRs #1, #5, #7).Two run modes, one line apart
docker-compose.yamlis the base: bucket storage, the write API, the PowerSync service. On its ownit is Adopter Mode — your database, no demo client. An overlay under
examples/adds a seededsource database, the demo client and its own PowerSync config, giving Example Mode.
Selection is the
COMPOSE_FILEline in.env, so the command stays a plaindocker compose upand
down,logsandpsall behave normally.Four flavours are bundled: Postgres, MongoDB, MySQL (Beta) and SQL Server (Beta). Each has its own
overlay, seed data, PowerSync config and README, and each was brought up from a dropped volume and
verified end to end — seeded rows synced, and a write round-tripped through the source database and
back. They differ where the databases differ: MySQL needs binlog settings mounted before the server
starts rather than applied afterwards, SQL Server needs a one-shot CDC bootstrap container and runs
under amd64 emulation, and MongoDB adds no container at all, using a second database on the replica
set bucket storage already runs.
Bucket storage always stays in a container this project owns. We never create schemas in a database
you merely pointed us at.
Also in here
A development overlay. Append
docker-compose.dev.yamlto any mode and your working tree ismounted in with the watch process running — an edit is serving in about two seconds, no rebuild.
Works in Adopter Mode too, which is where it matters most: wiring in your own database is exactly
when you are editing
src/persistance/andsrc/auth/verifier.ts.Sync rules move to the compiled-streams format (edition 3), renamed to
sync-config.yaml.Whatever ships here is what an adopter copies for their own schema, and upstream now documents only
the new shape.
A committed throwaway keypair. Without a stable key the backend mints a new one on every restart
and PowerSync rejects tokens it accepted moments earlier (
PSYNC_S2101). The keys in.envarepublic and marked as such;
pnpm generate-keysprints a replacement.Verification
Every mode was brought up and exercised by hand;
test.txtis the manual test sheet.AI disclosure
This PR was created with the help of Claude Code. Help constitutes assistance in research, planning,
and rough outline of implementation. Beyond having a hand in the implementation, I have also manually
tested this work.