Skip to content

Latest commit

 

History

History

test-version-utils

README.md

@fluid-private/test-version-utils

This is a package for writing and setting up Fluid end to end tests using mocha that will generate variants with a specific driver and different version combinations of Fluid API between layers via TestObjectProvider provided to the test. The different layers are loader, driver, container runtime and data runtime (includes DDS). Version combinations and driver selection can be controlled via the mocha command line, assuming your test uses the provided describe* functions. For advanced usage, a test can bypass this mechanism and directly call our exports to get the versioned Fluid APIs.

NOTE: This package is private to the @microsoft/fluid-framework repository. This package is not published. Use it only in packages in the same pnpm workspace. Specify workspace:* as the version. Use this package only as a development dependency or as a dependency of an unpublished package.

Versioned combination test generation

Layer version combinations

Similar to mocha's describe, this package provide a describeCompat function that will generate variants with a specific driver and different version combinations of Fluid API between layers. All possible layer combinations that are generated (empty entries are current versions):

Compat Kind Loader Driver Container Runtime Data Runtime
None
Loader old
LoaderDriver old old
Driver old
ContainerRuntime old
DataRuntime old
NewLoader old old old
NewDriver old old old
NewContainerRuntime old old old
NewDataRuntime old old old

Cross-Client combinations

In addition to the layer version combinations seen above, this package also provides functions to generate variations intended to test all layers of one version against all layers of another version in tests that feature more than one client. The intention is to simulate scenarios where the client that created a document was using a different version than the client loading the document. The matrix pairs the current build against every in-window prior Compatibility Checkpoint at or above the deployed-client compatibility floor (see CompatibilityCheckpoints.md) in both directions, with each prior checkpoint resolved to the earliest minor in its range (e.g. CC#3 → latest patch of 2.40.x).

The data driving the matrix lives in src/checkpoints.ts.

Mocha test setup with layer version combinations

describeCompat expects 3 arguments (name: string, compatVersionKind: CompatVersionKind, tests). There are three compatVersionKind options to generate different combinations, depending of the need of the tests:

FullCompat: generate test variants with compat combinations that varies the version for all layers.

  • Used for tests that exercise all layers and will benefits compat combinations of all layers.

LoaderCompat: generate test variants with compat combinations that only varies the loader version.

  • Use for tests that targets the loader layer, and don't care about compat combinations of other layers.
  • Test combination generated: [CompatKind.None, CompatKind.Loader]

This compat describe* function will also load the APIs with appropriate version and provide the test with a TestObjectProvider object, where the test can use to access Fluid functionality. Even when compat testing is not necessary, TestObjectProvider provide functionalities that help writing Fluid tests, and it allows the test to enable compat testing easily in the future just by changing the compatVersionKind parameter.

Legacy version defaults and installation

N is the base version under test (--baseVersion), normally the current repository build. N-1 and N-2 select one and two release bands before the band containing N, rather than subtracting a major, minor, or patch version. Stable release bands are bounded by successive multiples of ten in the minor version; not every intervening minor version is published. These boundaries match the designated minor releases where @beta and @legacy API breaking changes are permitted. See the release-tag stability guarantees and calculateRequestedRange in src/versionUtils.ts for the policy and selection rules, including transitions between major versions.

By default, describeCompat generates test variants for the following version configurations, subject to the suite's compatibility filter:

  • Non-compat: the current build across all layers.
  • N-1: the latest installed release in the previous release band, with all layer permutations.
  • Oldest Compatible Version (OCV): the exact oldestCompatibleVersion in src/compatConfig.ts, with an old loader or old loader plus driver and current other layers.
  • Cross-client: the current build paired with compatibility checkpoint versions as described above, resolved independently of OCV.

N-2 is part of the opt-in FULL mode, not the default matrix. FULL selects the latest installed release in each release band from N-2 down to the OCV floor, with old-loader and old-loader-plus-driver permutations.

Versioned packages are pre-installed in the committed compatibility workspace, and loaded by a Mocha setup hook.

Command line options

Tests using describeCompat will be controllable using the command line options when running mocha on driver selection, versions for compat testing, and compat kind combinations.

--compatKind <CompatKind> - filter to the compat variant. See above table. Can be specified multiple times.
                                Default: undefined (no filter)
--compatVersion <version> - specify the old version. Relative (to base) or specific version. Can be specified multiple times.
                                Default: 0, -1, OCV, CROSS_CLIENT (see src/compatConfig.ts)
--baseVersion <version>   - specify the base (new) version. Allow the test to run against any version combinations.
                                Default: <current> (same version of the test package)
--driver <driverType>     - <driverType> = "tinylicious" | "t9s" | "routerlicious" | "r11s" | "odsp" | "local"
                                Default: "local"
--r11sEndpointName <name> - Determine the environment variable name to look for r11s service information to target.
                                Default: "r11s"
--tenantIndex <number>    - Index into the tenant list, modulo the number of tenant available, for odsp server info to target.
                                Default: 0
--reinstall               - Force reinstall any required versions.  Default: reuse if a version is already installed.

baseVersion is a semver or a semver range.

  • If it is a semver range, the latest in that range will be picked.

compatVersion can be a semver or semver range or an integer apply to the minor version relative to baseVersion

The named modes are OCV (exact configured floor, loader and loader-plus-driver), FULL (stable back-compat release bands from N-2 down to the OCV floor, loader and loader-plus-driver), and CROSS_CLIENT (cross-client combinations described above). Use --compatVersion=FULL --compatVersion=OCV to include both stable release-band coverage and the exact boundary.

  • If it is a semver range, the latest in that range will be picked.
  • If it is an integer, the value will be apply to the minor version of the baseVersion, and create a ^ range including prerelease versions. The latest in that range will be picked.
    • i.e. if baseVersion is 0.2.3, and compatVersion is -1, the resulting range will be ^0.1.3-0

We also accept some of the flags via environment variables.

Setting fluid__test__backCompat=FULL adds the floor-bounded FULL configurations to the default matrix.

Environment Variables Command line options
fluidtestcompatKind --compatKind
fluidtestcompatVersion --compatVersion
fluidtestdriver --driver
fluidtestr11sEndpointName --r11sEndpointName
fluidtestbaseVersion --baseVersion

Advanced usage

This bypasses any configuration of version used by the describe* functions and provides direct access to the versioned APIs.

First make sure to call ensureVersionLoaded before running the tests to make sure the necessary legacy versions are available.

The main entry point is getVersionedTestObjectProvider to get a TestObjectProvider for a specific version combinations and driver config. Additionally, you can get versioned API for specific layers using these API.

  • getLoaderApi
  • getDriverApi
  • getContainerRuntimeApi
  • getDataRuntimeApi

All these API returns the current version by default if no arguments is passed. If a number is provided, a relative version will be computed by adding the number to the minor version number of the current version, and find the latest patch version. (^0.<current+requested>.0). If a string is provided, then the string is treated as a specific version or a range of version, and it will resolve the latest version that matches it.

OPEN ISSUE: while these API can be used directly, currently the default global mocha hook will still run and install the default set of legacy versions whether it is necessary or not.

Updating compat versions

After a Fluid Framework version bump or after a new compatibility checkpoint is designated, build package dependencies (pnpm exec fluid-build --task build:esm /test-version-utils$), and then run the following from this package's directory:

pnpm run update-compat-versions

The script (scripts/updateCompatVersions.ts) does the following:

  1. Reads the current package version from src/packageVersion.ts.
  2. Maps that version to a checkpoint via src/checkpoints.ts and queries the npm registry to resolve every in-window prior checkpoint, plus the full back-compat versions, to exact versions.
  3. Writes compat-workspaces/generated-versions.cjs with the resolved exact versions.
  4. Creates or updates per-version package.json files in compat-workspaces/full/.
  5. Removes version directories that are no longer needed.
  6. Runs pnpm install --no-frozen-lockfile in the workspace to regenerate the committed lockfile.

Commit all files produced by the script: generated-versions.cjs, per-version package.json files, and compat-workspaces/full/pnpm-lock.yaml.

Adding pinned versions for specific tests

When a test needs a pinned version outside the generated version range, add that version to compat-workspaces/explicit-versions.mjs. Include a comment indicating which test needs the version and why. Then re-run update-compat-versions and commit the changes.

Implementation notes

Legacy packages are installed in a committed pnpm sub-workspace at compat-workspaces/full/. The workspace is installed automatically when pnpm install is run from the repo root (via the postinstall hook in this package's package.json), so no runtime installation step is required in tests.

For now, the current versions are statically bound to also provide typings. This is a lie since the public API of a package may change over time: ContainerRuntime in FF@10.0.0 will not have the same public API as ContainerRuntime in FF@1.0.0.

For the most part, public API breaks are relatively contained and the type is "correct enough" that increasing the complexity of the typing setup isn't worth the associated redesign. However, this does give rise to several places in this package and test-utils that have intentional "back-compat" code. For example, test-utils/src/testContainerRuntimeFactory.ts adapts the loadRuntime registry signature and compatibility-setting property names across supported historical and current packages.

Thus it's important to keep in mind that the type provided by static import or import type might not align exactly with the runtime object once taking the compatibility configuration into account.

ChannelFactoryRegistry Rewriting

This package currently has some logic to rewrite the ChannelFactoryRegistry used to create a TestObjectProvider.

This means that statically importing and referencing a DDS in a test file will correctly result in referencing the version of that DDS defined in the compatibility configuration, but this happens implicitly. Test authors are encouraged to use the apis argument of describeCompat's test creation callback to reference the DDS instead.

Contribution Guidelines

You can contribute to Fluid Framework in these ways:

For detailed instructions, read the repo documentation.

This project follows the Microsoft Open Source Code of Conduct. For more information, read the Code of Conduct frequently asked questions. For questions or comments, contact opencode@microsoft.com.

This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services. Use of these trademarks or logos must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.

Help

Read the Fluid Framework documentation for information about Fluid Framework concepts and APIs.

To request information that the documentation does not contain, create an issue.

Trademark

This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services.

Use of these trademarks or logos must follow Microsoft's Trademark & Brand Guidelines.

Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.