Skip to content

Support custom ZooKeeper ACLs when creating clusters through helix-rest - #355

Merged
bellatrix007 merged 6 commits into
linkedin:devfrom
bellatrix007:adbansal/rest-cluster-acl
Oct 22, 2026
Merged

bellatrix007 merged 6 commits into
linkedin:devfrom
bellatrix007:adbansal/rest-cluster-acl

Conversation

@bellatrix007

@bellatrix007 bellatrix007 commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Issues

  • My PR addresses the following Helix issues and references them in the PR description:

Follow-up to #224 (see the review thread on HelixAdmin.java asking about ClusterSetup and the REST ClusterAccessor). There is no tracking issue.

Description

  • Here are some details about my PR, including screenshots of any UI changes:

#224 added HelixAdmin#addCluster(String, boolean, List<ACL>), but clusters created through helix-rest (PUT /clusters/{id}) still got the client default ACL. ClusterAccessor calls ClusterSetup, which only called the two-argument addCluster. This PR lets the caller of cluster creation pass an ACL through that path. helix-rest is a single shared deployment, so the ACL is supplied per request rather than through a custom launcher or a namespace manifest.

  • ClusterSetup: new addCluster(name, overwrite, cloudConfig, List<ACL>). The existing overloads delegate with a null ACL, so current callers are unchanged. The ACL is applied to the cluster root, the metadata nodes and the built-in state model definitions.
  • HelixAdmin#addStateModelDef(..., boolean recreateIfExists, List<ACL>): new overload. The default implementation delegates when the ACL is null or empty and throws UnsupportedOperationException otherwise; ZKHelixAdmin creates the definition node with the ACL. The existing overloads are unchanged.
  • ClusterAclParser (org.apache.helix.rest.acl): parses scheme:id:permissions entries into ACLs. Errors name the offending entry and show the expected format with examples.
  • PUT /clusters/{id} accepts repeatable, URL-encoded acl query parameters, documented in the method Javadoc. A malformed entry returns 400 before anything is created. world:anyone is always granted read in addition to the requested ACL (merged into a requested world:anyone entry). Without acl, the client default ACL is kept, which already allows read.

Example:

PUT /admin/v2/clusters/my-cluster?acl=x509%3ACN%3Dmy-service%2COU%3Dmy-team%2CO%3Dmy-org%2CC%3DUS%3Acdrw&acl=world%3Aanyone%3Ar

The scheme ends at the first colon and the permissions start after the last one, so ids containing commas (x509 DNs) or colons (digest) are kept intact. Permissions are any of c, d, r, w, a, or all. The auth scheme takes an empty id (auth::cdrwa); every other scheme requires one.

Caveats:

  • The REST server's own ZooKeeper identity must be granted CREATE by the ACL (and DELETE when recreate=true). Otherwise creation fails with a 500 and the partial state described in Support custom ZooKeeper ACLs when creating a cluster #224 can remain.
  • WRITE is only needed for the Azure cloud provider, which also updates the cluster config. The cloud config node itself is created with the client default ACL.
  • auth::perms expands to the authenticated identities of the REST server's ZooKeeper session. If that session is unauthenticated, ZooKeeper rejects it as an invalid ACL and the request returns 500.
  • The x509 id format depends on how the ensemble derives client ids from certificates. Stock ZooKeeper uses the subject DN; other deployments may derive a different id.
  • Digest ACLs cannot be used for the REST server's own session because RealmAwareZkClient has no addAuthInfo. Use x509 or SASL.
  • As in Support custom ZooKeeper ACLs when creating a cluster #224, only nodes created by addCluster and the built-in state model definitions carry the ACL. Resources and instances written later use the default ACL.
  • Ensembles that assign ACLs server side (setX509ClientIdAsAcl) ignore the supplied list, and ACLs have no effect with skipACL=yes.
  • The ClusterSetup command line is unchanged.
  • Request acl values are caller-controlled, so who may set them is governed by the existing authorization on cluster creation.

Tests

  • The following tests are written for this issue:

  • TestClusterAcl (helix-rest, embedded ZooKeeper): request acl parameters applied to the cluster root, metadata nodes and every built-in state model definition; multiple entries keep their order; read merged into a requested world:anyone entry; malformed entries return 400 and create nothing; no acl keeps the default; auth:: without an authenticated session returns 500 and creates nothing; a digest ACL that excludes the REST server returns 500.

  • TestClusterAclParser (helix-rest): permissions, ids with commas and colons, auth:: accepted and empty ids rejected for other schemes, unmodifiable result, malformed entries with the format hint, world-read append and merge.

  • TestClusterSetupWithAcl (helix-core): ACL passed through ClusterSetup to the root and every built-in state model definition, also on recreate; the ACL-aware addStateModelDef respects recreateIfExists, and an invalid or null definition is rejected before anything is deleted; null ACL keeps the default.

  • The following is the result of the "mvn test" command on the appropriate module:

Run locally on JDK 11 with mvn -o -pl helix-rest -am test -Dtest=... -Dcheckstyle.skip. All of these passed, with no failures or errors:

  • helix-rest: TestClusterAcl, TestClusterAclParser, TestClusterAccessor, TestHelixRestServer
  • helix-core: TestClusterSetupWithAcl, TestClusterSetup, TestZkHelixAdmin

The full helix-rest and helix-core suites were not run locally.

Also checked by hand earlier against a running HelixRestMain and a local ZooKeeper: curl -G -X PUT .../admin/v2/clusters/X --data-urlencode "acl=world:anyone:r" --data-urlencode "acl=ip:127.0.0.1:cdrwa" returned 201 and getAcl on the cluster znodes matched; acl=not-an-acl returned 400; no acl kept world:anyone:cdrwa. x509 ACLs were not exercised because that ZooKeeper has no x509 auth provider.

  • Local code review completed

Changes that Break Backward Compatibility (Optional)

  • None. The existing ClusterSetup#addCluster overloads keep their behavior; the new ACL path is only used when a request acl parameter is present. ClusterAccessor#createCluster gains one parameter, which only matters to callers invoking it directly rather than over HTTP. HelixAdmin#addStateModelDef gains a default method, so existing implementations still compile.

Documentation (Optional)

  • No wiki page added.

Commits

  • My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Some commit subjects are longer than 50 characters. Bodies wrap at 72. The branch has six commits; the third removes an earlier provider and namespace-manifest approach.

Code Quality

  • My diff has been formatted using helix-style.xml
    (helix-style-intellij.xml if IntelliJ IDE is used)

The formatter was not run; the code follows the style of the surrounding files.

🤖 Generated with GitHub Copilot CLI

Clusters created via PUT /clusters/{id} still got the client default ACL,
because ClusterAccessor -> ClusterSetup only called the two-argument
HelixAdmin#addCluster. Wire the ACL-aware overload added in linkedin#224 through
the REST path.

- ClusterSetup: add addCluster(name, overwrite, cloudConfig, List<ACL>);
  existing overloads delegate with a null ACL.
- ClusterAclProvider / NoopClusterAclProvider: pluggable per-request ACL
  source, passed to HelixRestServer through a new constructor overload.
- HelixRestNamespace: optional CLUSTER_ACL (list of scheme:id:perms),
  parsed by ClusterAclParser and read from the namespace manifest. It is
  not exposed through the namespaces endpoint.
- ClusterAccessor#createCluster resolves the ACL as provider, then
  namespace ACL, then the client default (unchanged behavior).

The REST server's own ZooKeeper identity must be granted CREATE by the
ACL (and WRITE when a cloud config is supplied). As in linkedin#224, only nodes
created by addCluster are covered.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Built-in state-model definitions remain world-writable despite a custom cluster ACL.

1 open finding
What changed in this PR

Extends REST cluster creation to support custom ZooKeeper ACLs from namespace manifests or request-aware providers.

Changes:

  • Propagates ACLs through ClusterAccessor and ClusterSetup.
  • Adds ACL provider, parser, namespace configuration, and server wiring.
  • Adds unit and integration coverage for parsing, precedence, failures, and ACL application.
File Description
helix-core/​src/​main/​java/​org/​apache/​helix/​tools/​ClusterSetup.java Adds ACL-aware cluster creation.
helix-core/​src/​test/​java/​org/​apache/​helix/​tools/​TestClusterSetupWithAcl.java Tests ACL propagation through ClusterSetup.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​acl/​ClusterAclParser.java Parses manifest ACL entries.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​acl/​ClusterAclProvider.java Defines request-aware ACL sourcing.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​acl/​NoopClusterAclProvider.java Provides backward-compatible fallback behavior.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​common/​ContextPropertyKeys.java Adds the provider context key.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​common/​HelixRestNamespace.java Stores namespace ACL configuration.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​server/​HelixRestMain.java Loads ACLs from YAML manifests.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​server/​HelixRestServer.java Injects the ACL provider into REST resources.
helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​server/​resources/​helix/​ClusterAccessor.java Resolves and applies ACLs during creation.
helix-rest/​src/​test/​java/​org/​apache/​helix/​rest/​acl/​TestClusterAclParser.java Tests ACL syntax and permissions.
helix-rest/​src/​test/​java/​org/​apache/​helix/​rest/​server/​TestClusterAcl.java Tests end-to-end REST ACL behavior.
helix-rest/​src/​test/​java/​org/​apache/​helix/​rest/​server/​TestHelixRestMainManifest.java Tests manifest ACL loading.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread helix-core/src/main/java/org/apache/helix/tools/ClusterSetup.java
PUT /clusters/{id} now takes repeatable acl query parameters, each a
scheme:id:permissions entry. They take precedence over the
ClusterAclProvider and the namespace CLUSTER_ACL, so callers of a
deployed REST server can set the ACL without a manifest or a custom
main. A malformed entry returns 400 before anything is created.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 14, 2026 05:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Built-in metadata remains open under restrictive ACLs, and valid ZooKeeper auth ACL entries are rejected.

1 open finding
Previously missed (3)

In code that hasn't changed since last review

Medium severity Permit empty IDs for ZooKeeper auth ACLs

helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​acl/​ClusterAclParser.java:75

This rejects ZooKeeper's valid auth::cdrwa form because the built-in auth scheme intentionally uses an empty ID. As a result, callers cannot configure an ACL bound to the REST server session's authenticated identities through either query parameters or the manifest. Permit an empty ID for the auth scheme while retaining the empty-ID rejection for other schemes, and add a parser case for it.

Low severity Restrict cloud config WRITE requirement to Azure

helix-core/​src/​main/​java/​org/​apache/​helix/​tools/​ClusterSetup.java:236

This overstates the required permission: addCloudConfig creates /CONFIGS/CLOUD/{cluster}; only the Azure branch below updates the existing cluster-config node. Saying every cloud config requires WRITE can cause operators to overgrant privileges. Limit this requirement to Azure cloud configuration (and make the corresponding ClusterAclProvider wording match).

Low severity Clarify Azure-only WRITE requirement for ACL providers

helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​acl/​ClusterAclProvider.java:34

The WRITE requirement is only triggered by the Azure topology update in ClusterSetup; other cloud configurations are written to a newly created cloud-config znode. Keep this public extension-point contract precise so provider implementers do not grant unnecessary write access.

🧠 Review effort: Balanced

helix-rest is a single shared deployment, so there is no custom launcher
to plug a ClusterAclProvider into and the default namespace cannot be
configured through a manifest. Keep only the acl query parameter on
cluster creation and remove the provider SPI, the namespace CLUSTER_ACL
property and the manifest parsing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 14, 2026 09:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Built-in state-model nodes remain openly writable, and valid auth::permissions ACLs are rejected.

2 open findings

🧠 Review effort: Balanced

When a create-cluster request carries acl parameters, add world:anyone
read to the ACL, merging it into a requested world:anyone entry. A
request without acl keeps the client default, which already allows
read. Malformed acl entries now return the offending entry and the
expected scheme:id:permissions format with examples, and the endpoint
Javadoc documents the parameter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 14, 2026 09:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Built-in state-model definitions remain open despite a requested custom ACL.

4 open findings

🧠 Review effort: Balanced

Comment thread helix-rest/src/main/java/org/apache/helix/rest/acl/ClusterAclParser.java Outdated
for (String path : new String[] {
"/" + cluster, PropertyPathBuilder.idealState(cluster),
PropertyPathBuilder.clusterConfig(cluster), PropertyPathBuilder.liveInstance(cluster),
PropertyPathBuilder.controller(cluster), PropertyPathBuilder.stateModelDef(cluster)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop asserts the ACL on the /STATEMODELDEFS parent but never on its child definition records (/STATEMODELDEFS/MasterSlave, /LeaderStandby, …). Those children aren't created by _admin.addCluster(...) / createZKPaths — they're written afterward by ClusterSetup.addCluster's addStateModelDef(...) loop, which goes through the data accessor with no ACL and therefore lands at the ZkClient default OPEN_ACL_UNSAFE (world:cdrwa — world-writable and world-ADMIN).

Because this assertion only checks the parent, the test passes while the behavior-governing state-model definitions remain world-writable. This masks the gap already flagged on ClusterSetup.java. Recommend asserting the ACL on at least one concrete definition, e.g. PropertyPathBuilder.stateModelDef(cluster) + "/MasterSlave", so the hole is caught here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d135f99. assertClusterAcl in TestClusterAcl and the helix-core TestClusterSetupWithAcl now assert the ACL on each built-in state model definition node, not only the /STATEMODELDEFS parent. This caught the real gap: those children previously kept the open default ACL. It is fixed in ClusterSetup/ZKHelixAdmin as described in the first thread. I also dropped the ip:127.0.0.1-only test case that failed on CI, because the runner's ZooKeeper session was not matched by that IP. The remaining integration cases include world:anyone so the test client can create and drop the cluster. Appending read to a non-world ACL is covered by TestClusterAclParser.

}
}
if (!worldEntryFound) {
result.add(new ACL(ZooDefs.Perms.READ, WORLD_ANYONE));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

withWorldRead unconditionally grants world:anyone READ — merged into an existing world entry here, or appended when none is present. For the default (no-acl) path this is harmless since the client default is already OPEN_ACL_UNSAFE. But when a caller supplies a deliberately restrictive ACL — say only x509:myservice:cdrwa — the resulting cluster metadata is still world-readable with no way to opt out. In other words, this endpoint can never create a cluster whose metadata is private to specific identities.

If that's intentional (e.g. Helix clients/controllers connecting under varied identities need broad read), that's defensible — but it's worth confirming it satisfies tenant security requirements, and considering whether read should instead be granted to a specific reader identity rather than world:anyone, or exposed as an opt-out.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this as is. Always granting world:anyone read is intentional. Helix participants, controllers and spectators connect under varied identities and need to read cluster metadata. Without an acl parameter the default ACL already grants world read, so this keeps the two paths consistent. Callers still control write, delete and admin access. If an opt-out is needed later it can be added as an explicit parameter, but I'd prefer not to add one without a concrete use case.

throw new IllegalArgumentException(
"Invalid ACL entry \"" + entry + "\". " + ACL_FORMAT_HINT);
}
String scheme = entry.substring(0, schemeEnd);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parser validates the format (non-empty scheme/id/perms, valid perm chars) but never the scheme itself. An unknown or malformed scheme such as foo:bar:r passes parsing cleanly, then fails server-side at setACL with InvalidACL — surfacing as a 500 with partial, un-rolled-back cluster state (cf. testServerOutsideAclFailsCreation) instead of a clean 400.

Since the goal here is to reject bad input before anything is created, consider validating scheme against the schemes the ensemble actually supports (world/auth/digest/ip/x509/sasl) up front, so a typo'd scheme is caught at the 400 boundary rather than leaving a half-created cluster behind.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this as is. A scheme whitelist would reject ensemble-specific providers (x509, sasl, custom ones), so it would need to be configured per ensemble. An unsupported scheme or malformed id fails on the first node create with InvalidACL. That returns 500 and leaves no partial cluster; I confirmed this locally with x509 on an ensemble without that provider. Format errors the parser can detect, such as a bad shape, missing id or bad permission letters, are rejected up front with 400 and the expected format in the message.

…iew feedback

- Create built-in state model definition nodes with the requested ACL. ZooKeeper does not
  inherit ACLs, so they previously kept the open default ACL. Adds an ACL-aware
  HelixAdmin#addStateModelDef overload (default throws for non-empty ACLs, ZKHelixAdmin
  implements it) used by ClusterSetup#addCluster.
- Accept the auth scheme with an empty id (auth::perms); other schemes still require an id.
- Use a generic x509 DN in the ACL format hint and Javadoc.
- Document that WRITE is only needed for the Azure cloud config, and that the cloud config
  node keeps the default ACL.
- Make the REST ACL integration tests independent of the runner's IP, assert ACLs on every
  built-in state model definition, and cover auth without an authenticated session.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 14, 2026 14:29
@bellatrix007

Copy link
Copy Markdown
Collaborator Author

Addressing the review note on cloud config in d135f99: the ClusterSetup#addCluster Javadoc and PR description no longer say a cloud config always needs WRITE. addCloudConfig only creates the cloud config node, which uses the client default ACL. WRITE is needed only for the Azure provider, which also updates the cluster config.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Invalid state-model recreation can delete an existing definition before validation, and the REST documentation misstates cloud-config ACL coverage.

1 open finding
4 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Low severity Document cloud-config znode ACL exception

helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​server/​resources/​helix/​ClusterAccessor.java:186

This security boundary is inaccurate when addCloudConfig=true: ClusterSetup creates the cloud-config znode through the default-ACL path, not with the requested ACL. Document that exception here so REST callers do not assume all nodes created by this request are protected.

🧠 Review effort: Balanced

Comment thread helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java Outdated
}

if (acl != null && !acl.isEmpty()) {
if (stateModel == null || !stateModel.isValid()) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recreateIfExists deletes the existing definition before this validation runs.

When recreateIfExists == true and a non-empty acl is supplied, the node at stateModelPath has already been removed by _zkClient.deleteRecursively(stateModelPath) in the exists/recreateIfExists block above by the time we reach this stateModel == null || !stateModel.isValid() check. A null or invalid model therefore destroys the existing (valid) definition and then throws, leaving the cluster with no definition at that path.

This isn't reachable through the REST path (built-in defs are always valid), but it's a real data-loss ordering bug for any direct caller of this ACL-aware overload. Suggest validating before the destructive delete — e.g. hoist the null/validity check above the exists block, while keeping the early return for the recreateIfExists == false case.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9f01e87. For a non-empty ACL, the null/validity check now runs before the exists/recreateIfExists block, so a rejected call no longer deletes the existing definition. The early return for recreateIfExists == false is unchanged, and so is the null/empty-ACL path. Added testInvalidStateModelDefWithAclDoesNotDeleteExistingDefinition: a recreate with an invalid or null model now throws and leaves the existing definition and its ACL intact. I confirmed that test fails without the fix.

With a non-empty ACL, addStateModelDef deleted the existing definition
when recreateIfExists was true and only then rejected a null or invalid
model, leaving the cluster without that definition. Validate first so a
rejected call leaves the existing definition and its ACL untouched.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 12:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Authorization-sensitive changes warrant human validation against authenticated x509 or SASL deployments not exercised by the reported tests.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Include offending ACL entry in permission validation errors

helix-rest/​src/​main/​java/​org/​apache/​helix/​rest/​acl/​ClusterAclParser.java:111

For an entry such as ip:127.0.0.1:rx, parsePermissions reports only Invalid ACL permissions "rx". The HTTP 400 response therefore omits the offending entry, unlike structural-validation errors and the documented behavior. With repeatable acl parameters, this makes the invalid entry harder to locate. Include the complete entry when propagating permission-validation errors.

1 resolved since last review

Review effort: Balanced

@sjainit sjainit left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@bellatrix007
bellatrix007 merged commit 2cfe93e into linkedin:dev Oct 22, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants