RDSC-5610: Multi-source documentation by stoyanr · Pull Request #3916 · redis/docs

stoyanr · 2026-09-03T13:21:30Z

See https://redislabs.atlassian.net/browse/RDSC-5610

Enhances #3841 with numerous changes and additions. I couldn't update the original PR since I don't have write permissions to this repository.

Note that it now documents 2 PRs that are not yet merged, but should be before the release:

Note

Low Risk
Documentation-only changes describing RDI configuration and operations; no application code or runtime behavior is modified in this PR.

Overview
Documents multi-source RDI pipelines: one pipeline can ingest from several source databases (mixed types) into a single Redis target, with a per-source CDC collector, credentials, and change streams. Adds Multiple sources in one pipeline covering source naming, config.yaml, secrets, jobs (server_name), add/remove sources, --source for start/stop/reset, monitoring/DLQ keys, and legacy naming after upgrade.

Secrets and deploy are rewritten around database-scoped keys (USERNAME, PASSWORD, TLS keys) and redis-di set-secret … --db <source|target>, with env vars like ${MYSQL_DB_PASSWORD}; legacy SOURCE_DB_* / TARGET_DB_* remain only for pre-multi-source single sources. K8s examples use <source>-db secrets and per-source collector metrics paths.

Product positioning updates: “when to use” and architecture pages say one or more sources; the decision tree drops the “single source only” branch. API v2 is required for multi-source; migration guide notes v1 limits. Flink jobs can match multiple sources/tables via lists and regex:.

Prepare-database guides, pipeline config examples, observability, HA testing, and FAQ are aligned with the new model.

^{Reviewed by Cursor Bugbot for commit 5dc2f92. Bugbot is set up for automated code reviews on this repo. Configure here.}

andy-stark-redis

Quite a few suggestions but they are easily applied, so I'll approve to avoid delays.

…vs-flink.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

…e-dbs/_index.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

…e-dbs/snowflake.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

…e-dbs/spanner.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

ZdravkoDonev-redis · 2026-09-08T12:45:18Z

  source:
    mongodb.ssl.enabled: true
-    mongodb.ssl.keystore: /debezium/certs/source_db_keystore
+    mongodb.ssl.keystore: /debezium/certs/mongodb_db_keystore


@stoyanr As we discussed, lets remove this property altogether in the public docs after we have it.

ZdravkoDonev-redis · 2026-09-08T12:57:04Z

+| Credentials secret | `mysql-db` |
+| TLS secret | `mysql-db-ssl` |
+| Environment variable prefix | `MYSQL_DB_` |
+| Certificate directory | `/etc/certificates/mysql_db/` |


@stoyanr Similarly here, I think we should remove this from the docs?

This one should stay. These certificate directories, although internal, are referred to elsewhere (if setting secrets manually). Their naming is not affected by the keystore properties, they are valid on all databases even if they don't use a keystore. I would only remove them if we decide to stop documenting the manual K8s secret setup.

…e-dbs/mongodb.md Co-authored-by: Zdravko Donev <zdravko.donev@redis.com>

zhivko-kirishev-redis

Everything about the automatic population of the ssl properties as well as the default Flink processor looks good

nicoweidner

I left a few comments. We can tackle them in a followup so as not to block the release, none of them are critical

nicoweidner · 2026-09-09T20:53:48Z

-
-```text
-dlq:data:{rdi}:public.users
+dlq:data:{rdi}:<source>.<schema_or_database>.<table>


Is this form true for SQLServer and Snowflake? IIRC from some review those two were exceptions and had a three-part fully qualified table name even without the source component

nicoweidner · 2026-09-09T21:04:50Z

+
+{{< note >}}The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be
+used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with
+exactly one source created before RDI supported multiple sources. See


I don't think the "created before RDI supported multiple sources" part is correct, we'll accept it as long as the pipeline has a single source. But maybe we can add a note that they are considered deprecated, to discourage their use?

nicoweidner · 2026-09-10T06:44:37Z

+| Environment variable prefix | `MYSQL_DB_` |
+| Certificate directory | `/etc/certificates/mysql_db/` |
+| Collector deployment | `collector-mysql` |
+| Change data streams | `data:{rdi}:mysql.<schema_or_database>.<table>` |


This is incorrect for SQLServer which uses data:{rdi}:<source>.<database>.<schema>.<table>. Same for DLQ keys. We could make it more universal by writing dlq:data:{rdi}:<source>.<qualified_table_name>, wdyt?

nicoweidner · 2026-09-10T06:47:51Z

+the reset runs and starts it again afterwards, exactly as it does for a reset of the whole
+pipeline.
+
+Data that is not partitioned by source is never deleted by a per-source reset.


I don't understand this sentence. Does it refer to any metadata in the RDI DB that is not per-source data?

I'd suggest clarifying which data we mean here, or potentially dropping it entirely, as the previous paragraph already lists which data is deleted.

nicoweidner · 2026-09-10T06:49:12Z

+[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}).
+
+Dead-letter queue streams have Redis keys containing a
+`<source>.<schema_or_database>.<table>` section.


Same comment regarding the key format for SQLServer applies here

nicoweidner · 2026-09-10T06:54:43Z

+    schemas:
+      - public
+    tables:
+      public.customers: {}


A review agent found that this example has a potential target key collision, since the default key generator will use e.g. customers:id:1 if the PK is id for both sources. While I think this misses the point of the example, we could still clear any potential confusion by renaming the second table to something else like public.clients

nicoweidner · 2026-09-10T06:59:56Z

+When a pipeline has more than one source, every job must set `server_name`, and the value
+must match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no


An aliased source retains server_name: rdi; we should probably point out this exception here. There is also Spanner, which is an exception to the exception and used its instance id before the upgrade. Given the limited number of Spanner customers and the fact that this additional exception only becomes relevant when adding a second source to a legacy pipeline, we could consider omitting this special case from the docs

nicoweidner · 2026-09-10T07:52:33Z

+- Its data streams are still named `data:{rdi}:<schema_or_database>.<table>`, and its offset
+  and schema history keys are still `metadata:debezium:offsets` and
+  `metadata:debezium:schema_history`.
+- Its `server_name` is still `rdi`.


... except if it's Spanner, then it would be the instance id.

nicoweidner · 2026-09-10T09:57:25Z

+  table: customers
+```
+
+When a pipeline has more than one source, every job must set `server_name`, and the value


Suggested change

When a pipeline has more than one source, every job must set `server_name`, and the value

When a pipeline has more than one source, every job (apart from the default job) must set `server_name`, and the value

nicoweidner · 2026-09-10T10:08:50Z

+pipeline status is not broken down per source. RDI reports the whole pipeline in an error state when a
+single source fails, so you should use the `Components` section to find out which one has failed.
+
+Each source's collector has its own metric collection, named after the collector, such as


We don't have collections for Spanner and Snowflake right now.
Also, the incoming entries and latency metrics described here are for the classic processor. Since the Flink processor becomes the default now, we should probably link to its metrics instead of the classic processor

RDSC-5610: Multi-source documentation

ead6efa

mich-elle-luna requested review from ZdravkoDonev-redis and andy-stark-redis September 4, 2026 14:14

andy-stark-redis approved these changes Sep 8, 2026

View reviewed changes

stoyanr and others added 15 commits September 8, 2026 14:16

Update content/integrate/redis-data-integration/architecture/classic-…

3f20f01

…vs-flink.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Update content/integrate/redis-data-integration/data-pipelines/prepar…

4a7feba

…e-dbs/_index.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Update content/integrate/redis-data-integration/data-pipelines/prepar…

efa626e

…e-dbs/snowflake.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Update content/integrate/redis-data-integration/data-pipelines/prepar…

70ad799

…e-dbs/snowflake.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Update content/integrate/redis-data-integration/data-pipelines/prepar…

f5cb4db

…e-dbs/spanner.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Update content/integrate/redis-data-integration/data-pipelines/prepar…

96dab60

…e-dbs/spanner.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

b926728

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

559a75c

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

72ac00c

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

edec338

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

1daabdb

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

9e8bb08

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

af504e9

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

3aacb11

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Apply batched suggestions from code review

6175be4

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

ZdravkoDonev-redis reviewed Sep 8, 2026

View reviewed changes

Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md Outdated

ZdravkoDonev-redis reviewed Sep 8, 2026

View reviewed changes

stoyanr and others added 4 commits September 8, 2026 16:11

Update content/integrate/redis-data-integration/data-pipelines/prepar…

38c8d28

…e-dbs/mongodb.md Co-authored-by: Zdravko Donev <zdravko.donev@redis.com>

Update per-source reset and deletion

a26c434

Update for changes introduced with RDSC-5882

5dc2f92

Update for making the Flink processor the default

79ab2f7

zhivko-kirishev-redis approved these changes Sep 9, 2026

View reviewed changes

nicoweidner reviewed Sep 10, 2026

View reviewed changes

kaitlynmichael added the rdi label Sep 11, 2026

		When a pipeline has more than one source, every job must set `server_name`, and the value
		must match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no

Uh oh!

Conversation

stoyanr commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andy-stark-redis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhivko-kirishev-redis left a comment

Choose a reason for hiding this comment

Uh oh!

nicoweidner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

stoyanr commented Sep 3, 2026 •

edited

Loading