This is a package for writing and setting up Fluid end to end tests using mocha that will generate variants with
a specific driver and different version combinations of Fluid API between layers via TestObjectProvider provided
to the test. The different layers are loader, driver, container runtime and data runtime (includes DDS). Version
combinations and driver selection can be controlled via the mocha command line, assuming your test uses the provided
describe* functions. For advanced usage, a test can bypass this mechanism and directly call our
exports to get the versioned Fluid APIs.
NOTE: This package is private to the @microsoft/fluid-framework repository.
This package is not published.
Use it only in packages in the same pnpm workspace.
Specify workspace:* as the version.
Use this package only as a development dependency or as a dependency of an unpublished package.
Similar to mocha's describe, this package provide a describeCompat function that will generate variants with
a specific driver and different version combinations of Fluid API between layers. All possible layer combinations that
are generated (empty entries are current versions):
| Compat Kind | Loader | Driver | Container Runtime | Data Runtime |
|---|---|---|---|---|
| None | ||||
| Loader | old | |||
| LoaderDriver | old | old | ||
| Driver | old | |||
| ContainerRuntime | old | |||
| DataRuntime | old | |||
| NewLoader | old | old | old | |
| NewDriver | old | old | old | |
| NewContainerRuntime | old | old | old | |
| NewDataRuntime | old | old | old |
In addition to the layer version combinations seen above, this package also provides functions to generate variations
intended to test all layers of one version against all layers of another version in tests that feature more than one client.
The intention is to simulate scenarios where the client that created a document was using a different version than the client
loading the document. The matrix pairs the current build against every in-window prior Compatibility Checkpoint
at or above the deployed-client compatibility floor (see
CompatibilityCheckpoints.md) in both directions, with
each prior checkpoint resolved to the earliest minor in its range (e.g. CC#3 → latest patch of
2.40.x).
The data driving the matrix lives in src/checkpoints.ts.
describeCompat expects 3 arguments (name: string, compatVersionKind: CompatVersionKind, tests). There are three compatVersionKind options to generate different combinations, depending of the need of the tests:
FullCompat: generate test variants with compat combinations that varies the version for all layers.
- Used for tests that exercise all layers and will benefits compat combinations of all layers.
LoaderCompat: generate test variants with compat combinations that only varies the loader version.
- Use for tests that targets the loader layer, and don't care about compat combinations of other layers.
- Test combination generated: [CompatKind.None, CompatKind.Loader]
This compat describe* function will also load the APIs with appropriate version and provide the test with a
TestObjectProvider object, where the test can use to access Fluid functionality. Even when compat testing
is not necessary, TestObjectProvider provide functionalities that help writing Fluid tests, and it allows the test
to enable compat testing easily in the future just by changing the compatVersionKind parameter.
N is the base version under test (--baseVersion), normally the current repository build.
N-1 and N-2 select one and two release bands before the band containing N, rather than subtracting
a major, minor, or patch version. Stable release bands are bounded by successive multiples of ten in the minor
version; not every intervening minor version is published. These boundaries match the designated
minor releases where @beta and @legacy API breaking changes are permitted.
See the release-tag stability guarantees
and calculateRequestedRange in src/versionUtils.ts for the policy and selection rules,
including transitions between major versions.
By default, describeCompat generates test variants for the following version configurations, subject to the suite's compatibility filter:
- Non-compat: the current build across all layers.
- N-1: the latest installed release in the previous release band, with all layer permutations.
- Oldest Compatible Version (OCV): the exact
oldestCompatibleVersioninsrc/compatConfig.ts, with an old loader or old loader plus driver and current other layers. - Cross-client: the current build paired with compatibility checkpoint versions as described above, resolved independently of OCV.
N-2 is part of the opt-in FULL mode, not the default matrix. FULL selects the latest installed release
in each release band from N-2 down to the OCV floor, with old-loader and old-loader-plus-driver permutations.
Versioned packages are pre-installed in the committed compatibility workspace, and loaded by a Mocha setup hook.
Tests using describeCompat will be controllable using the command line options when running mocha on
driver selection, versions for compat testing, and compat kind combinations.
--compatKind <CompatKind> - filter to the compat variant. See above table. Can be specified multiple times.
Default: undefined (no filter)
--compatVersion <version> - specify the old version. Relative (to base) or specific version. Can be specified multiple times.
Default: 0, -1, OCV, CROSS_CLIENT (see src/compatConfig.ts)
--baseVersion <version> - specify the base (new) version. Allow the test to run against any version combinations.
Default: <current> (same version of the test package)
--driver <driverType> - <driverType> = "tinylicious" | "t9s" | "routerlicious" | "r11s" | "odsp" | "local"
Default: "local"
--r11sEndpointName <name> - Determine the environment variable name to look for r11s service information to target.
Default: "r11s"
--tenantIndex <number> - Index into the tenant list, modulo the number of tenant available, for odsp server info to target.
Default: 0
--reinstall - Force reinstall any required versions. Default: reuse if a version is already installed.
baseVersion is a semver or a semver range.
- If it is a semver range, the latest in that range will be picked.
compatVersion can be a semver or semver range or an integer apply to the minor version relative to baseVersion
The named modes are OCV (exact configured floor, loader and loader-plus-driver), FULL (stable back-compat release
bands from N-2 down to the OCV floor, loader and loader-plus-driver), and CROSS_CLIENT (cross-client combinations described above).
Use --compatVersion=FULL --compatVersion=OCV to include both stable release-band coverage and the exact boundary.
- If it is a semver range, the latest in that range will be picked.
- If it is an integer, the value will be apply to the minor version of the
baseVersion, and create a^range including prerelease versions. The latest in that range will be picked.- i.e. if
baseVersionis0.2.3, andcompatVersionis-1, the resulting range will be^0.1.3-0
- i.e. if
We also accept some of the flags via environment variables.
Setting fluid__test__backCompat=FULL adds the floor-bounded FULL configurations to the default matrix.
| Environment Variables | Command line options |
|---|---|
| fluidtestcompatKind | --compatKind |
| fluidtestcompatVersion | --compatVersion |
| fluidtestdriver | --driver |
| fluidtestr11sEndpointName | --r11sEndpointName |
| fluidtestbaseVersion | --baseVersion |
This bypasses any configuration of version used by the describe* functions and provides direct access to the versioned APIs.
First make sure to call ensureVersionLoaded before running the tests to make sure the necessary legacy versions are available.
The main entry point is getVersionedTestObjectProvider to get a TestObjectProvider for a specific version combinations
and driver config. Additionally, you can get versioned API for specific layers using these API.
getLoaderApigetDriverApigetContainerRuntimeApigetDataRuntimeApi
All these API returns the current version by default if no arguments is passed.
If a number is provided, a relative version will be computed by adding the number to the minor version number
of the current version, and find the latest patch version. (^0.<current+requested>.0).
If a string is provided, then the string is treated as a specific version or a range of version, and it will
resolve the latest version that matches it.
OPEN ISSUE: while these API can be used directly, currently the default global mocha hook will still run and install the default set of legacy versions whether it is necessary or not.
After a Fluid Framework version bump or after a new compatibility checkpoint is designated, build package dependencies (pnpm exec fluid-build --task build:esm /test-version-utils$), and then run the following from this package's directory:
pnpm run update-compat-versions
The script (scripts/updateCompatVersions.ts) does the following:
- Reads the current package version from
src/packageVersion.ts. - Maps that version to a checkpoint via
src/checkpoints.tsand queries the npm registry to resolve every in-window prior checkpoint, plus the full back-compat versions, to exact versions. - Writes
compat-workspaces/generated-versions.cjswith the resolved exact versions. - Creates or updates per-version
package.jsonfiles incompat-workspaces/full/. - Removes version directories that are no longer needed.
- Runs
pnpm install --no-frozen-lockfilein the workspace to regenerate the committed lockfile.
Commit all files produced by the script: generated-versions.cjs, per-version package.json files, and
compat-workspaces/full/pnpm-lock.yaml.
When a test needs a pinned version outside the generated version range, add that version to compat-workspaces/explicit-versions.mjs.
Include a comment indicating which test needs the version and why.
Then re-run update-compat-versions and commit the changes.
Legacy packages are installed in a committed pnpm sub-workspace at compat-workspaces/full/.
The workspace is installed automatically when pnpm install is run from the repo root (via the
postinstall hook in this package's package.json), so no runtime installation step is required
in tests.
For now, the current versions are statically bound to also provide typings.
This is a lie since the public API of a package may change over time: ContainerRuntime in FF@10.0.0 will not have the
same public API as ContainerRuntime in FF@1.0.0.
For the most part, public API breaks are relatively contained and the type is "correct enough" that increasing the
complexity of the typing setup isn't worth the associated redesign.
However, this does give rise to several places in this package and test-utils that have intentional "back-compat" code.
For example, test-utils/src/testContainerRuntimeFactory.ts adapts the loadRuntime registry signature
and compatibility-setting property names across supported historical and current packages.
Thus it's important to keep in mind that the type provided by static import or import type might not align exactly with the
runtime object once taking the compatibility configuration into account.
This package currently has some logic to rewrite the ChannelFactoryRegistry used to create a TestObjectProvider.
This means that statically importing and referencing a DDS in a test file will correctly result in referencing the version of that DDS defined in the compatibility configuration,
but this happens implicitly.
Test authors are encouraged to use the apis argument of describeCompat's test creation callback to reference the DDS instead.
You can contribute to Fluid Framework in these ways:
- Answer questions in GitHub Discussions.
- Submit bug reports and help verify fixes.
- Review source code changes.
- Contribute bug fixes.
For detailed instructions, read the repo documentation.
This project follows the Microsoft Open Source Code of Conduct. For more information, read the Code of Conduct frequently asked questions. For questions or comments, contact opencode@microsoft.com.
This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services. Use of these trademarks or logos must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Read the Fluid Framework documentation for information about Fluid Framework concepts and APIs.
To request information that the documentation does not contain, create an issue.
This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services.
Use of these trademarks or logos must follow Microsoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.