[EvanDietzMorris]

Performance improvements + new Helm chart

This PR moves heavy processing out of the web server and makes Celery tasks lighter and more predictable. It simplifies how concurrency is controlled, which lets us remove Redis entirely. It also adds a Helm chart that deploys the whole ARS stack on Kubernetes.

---

1. ARA responses are processed in Celery, not in the web request

Before: when an ARA posted results back, the web server did all the work before responding. It parsed the full message, ran pre-merge processing and validation, compressed it, and then sent the whole message back in the response. ARA responses can be tens of MB, so a single large one could tie up a web worker for a long time. Many kinds of errors can also occur during that process, causing the ARA to see a dropped connection and retried, which added even more load.

Now: the endpoint stores the raw response (compressed, never parsed), queues a new ingest_ara_response Celery task, and returns a small acknowledgement right away. The celery task then does the processing the web server used to do.

Why it matters:

• A JSON message takes up several times its raw size in memory once it's parsed, and the web server handles many responses at once. With that work moved off the web server, its memory use depends far less on how big ARA responses are.
• The old path also did the same work twice. It parsed and compressed the message, then decompressed and re-serialized all of it to build a response the ARA doesn't use. Now the body is compressed once, as received.
• The web server acknowledges right away, so ARAs no longer time out and retry. That removes a source of duplicate load.

What ARAs and subscribers will notice:

• Validation failures are no longer returned as a 422 response. The child message is marked as an error and an ara_failed_validation notification is sent.
• ara_response_complete is now sent by the worker, so it arrives slightly later.
• A response body that can't be decoded as a JSON object (malformed JSON, an empty body, or a non-object such as {} or a list) used to get a 500 back, or for {}/lists end up Done with 0 results. The ARA now gets a 201, since the endpoint no longer parses the body, and the child message is marked as an error with code 400.

1a. New received_at timestamp

The callback endpoint now also records the moment an ARA's response arrives, before storing it, so that:

• Timeouts only measure the ARA call. A message stays "running" until the worker finishes with it, which now includes waiting in the queue. Without this, a response that arrived well within the 5 minute limit could still be timed out because our own queue was busy. catch_timeout no longer applies the 5 minute limit to messages that already have a response. Those get 20 minutes from received_at to finish processing instead, so a lost or crashed ingest task can't leave the query running forever. If the ingest task runs after its message has already timed out, it does nothing, so a timed-out message can't flip back to Done and get merged.
• Duplicate responses are rejected. Only the first callback for a pk can set the timestamp, in a single conditional update, so a second copy can't overwrite the stored response or be processed twice. The same update also refuses messages that are already Done or Error, so a response that arrives just as its message times out is rejected rather than accepted and then dropped. Previously the checks for this relied on fields that are only written once processing finishes.

2. Tasks pass IDs instead of whole messages

Before: the merge task received the entire message as its argument. That means the full message went through RabbitMQ, and went through again every time the task retried.

Now: the task receives only IDs and loads the message from the database itself. Tasks are queued only after the database write commits, so a worker always finds the data it needs.

Why it matters:

• Each queued task shrinks from megabytes to a few bytes. This matters most for retries, since a busy query can retry the same merge many times.
• The message is decompressed before the merge lock is taken rather than while holding it. Other merges on the same query spend less time waiting.

3. Clearer merge retry behavior

Building off of PR #860.

Before: two very different situations shared one small retry limit. One is waiting for another merge on the same query to finish, which is normal. The other is a real error. When the limit ran out, the merge gave up silently.

Now: each situation has its own limit, and both are configurable:

• Waiting on another merge: short, steady retries (ARS_MERGE_LOCK_MAX_RETRIES, default 100).
• Real errors: retries with increasing delays, so downstream services have time to recover (ARS_MERGE_ERROR_MAX_RETRIES, default 8).

If a merge gives up waiting, subscribers get a merged_version_failed notification.

4. Worker queues replace the Redis token gate, and Redis is removed

Before: a custom Redis-based token system limited how many expensive merges could run at once. It added a lot of moving parts. It only covered merges, not other heavy work, and in practice did not function well as intended. It could also cause messages to be sent through the rabbitMQ many more times than needed.

Now: heavy tasks (merging and ingesting ARA responses) go to a dedicated heavy Celery queue. Everything else stays on the default queue. The number of heavy workers sets the limit on concurrent heavy work, and quick tasks like notifications never wait behind a merge.

Why it matters:

• Predictable memory: each heavy worker runs a fixed number of tasks at a time. Its memory can be sized for that many large merges, instead of being a guess shared with every other kind of task.
• Bursts wait in line: when many responses arrive at once, the extra work waits in RabbitMQ until a worker is free. It doesn't keep retrying.
• Quick tasks stay quick: notifications and timeout checks have their own workers, so they stay responsive even when merges are backed up, without relying on the more complicated and harder to tune REDIS-based retry behavior.

The token gate was the only thing using Redis. The websocket/Channels setup that also referenced Redis was never enabled. So this PR removes Redis entirely: the gate code, the Channels settings and stub files, the Redis service in docker-compose, and the related dependencies.

5. Only the shepherd ARAs are active

Includes PR #880 - All non-shepherd ARAs and KPs are now listed as inactive in config/config.yaml. This PR also fixes a bug where that list was ignored for actors that already existed in the database.

6. Less tracing noise

Routine status checks and the health ping task are no longer traced.

---

7. New Helm chart (helm/ars/)

Before: the deploy/ chart runs the web server, Celery worker, RabbitMQ and Redis as containers in a single pod. That design has several problems:

• Nothing can scale. A second copy of the pod would also start a second broker.
• RabbitMQ has no persistent storage, so queued tasks are lost whenever the pod restarts.
• The chart doesn't include the mysql database, and the app's settings file is injected by Jenkins from outside the repo.

Now: helm/ars/ deploys the full stack, with each component in its own pod:

• Web server, heavy workers, light workers and Celery beat can each be scaled on their own.
• MySQL and RabbitMQ run in the cluster with persistent storage. Either one can point to an external service instead (for example, RDS).
• Configuration and credentials come from values and a Kubernetes Secret. No outside files are needed.
• Database migrations run as a separate Job, so multiple server replicas don't conflict.
• Health checks and startup ordering are included.
• Independent scaling: add web server replicas for more concurrent requests, or heavy worker replicas for more merge throughput.
• Memory doesn't creep: worker processes restart after a set number of tasks. One very large message doesn't keep a worker's memory high for the rest of its life.
• Faster database MySQL can get a larger memory cache, so reading and writing large stored messages doesn't constantly go to disk.

The chart comes with two environment overlays:

• renci-values.yaml: RENCI Sterling, with nginx ingress, TLS and sizing for the heavy workers and database.
• ncats-values.yaml: NCATS / transltr.io, with ECR images, ALB ingress, external RDS and node pinning.

See helm/ars/README.md for setup.

The legacy deploy/ chart is left in place.

---

Deployment notes

• Every Celery setup must consume the heavy queue, or merges will stall. The Helm chart handles this. Single-worker setups (docker-compose, the legacy chart, CI) now run with -Q celery,heavy.
• Deploy when the task queue is empty. Merge tasks queued by the old code are not compatible with the new task arguments.
• Configuration: new env vars are ARS_MERGE_LOCK_MAX_RETRIES and ARS_MERGE_ERROR_MAX_RETRIES. Removed env vars are ARS_EXPENSIVE_* and REDIS_URL.

Tests

New unit tests in tr_sys/tests/unit/ cover merge retries, ARA response ingestion and the callback endpoint. They need no database or broker. Run them from tr_sys/ with python -m pytest tests/unit.