You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support custom ZooKeeper ACLs when creating clusters through helix-rest - #355
My PR addresses the following Helix issues and references them in the PR description:
Follow-up to #224 (see the review thread on HelixAdmin.java asking about ClusterSetup and the REST ClusterAccessor). There is no tracking issue.
Description
Here are some details about my PR, including screenshots of any UI changes:
#224 added HelixAdmin#addCluster(String, boolean, List<ACL>), but clusters created through helix-rest (PUT /clusters/{id}) still got the client default ACL. ClusterAccessor calls ClusterSetup, which only called the two-argument addCluster. This PR lets the caller of cluster creation pass an ACL through that path. helix-rest is a single shared deployment, so the ACL is supplied per request rather than through a custom launcher or a namespace manifest.
ClusterSetup: new addCluster(name, overwrite, cloudConfig, List<ACL>). The existing overloads delegate with a null ACL, so current callers are unchanged. The ACL is applied to the cluster root, the metadata nodes and the built-in state model definitions.
HelixAdmin#addStateModelDef(..., boolean recreateIfExists, List<ACL>): new overload. The default implementation delegates when the ACL is null or empty and throws UnsupportedOperationException otherwise; ZKHelixAdmin creates the definition node with the ACL. The existing overloads are unchanged.
ClusterAclParser (org.apache.helix.rest.acl): parses scheme:id:permissions entries into ACLs. Errors name the offending entry and show the expected format with examples.
PUT /clusters/{id} accepts repeatable, URL-encoded acl query parameters, documented in the method Javadoc. A malformed entry returns 400 before anything is created. world:anyone is always granted read in addition to the requested ACL (merged into a requested world:anyone entry). Without acl, the client default ACL is kept, which already allows read.
Example:
PUT /admin/v2/clusters/my-cluster?acl=x509%3ACN%3Dmy-service%2COU%3Dmy-team%2CO%3Dmy-org%2CC%3DUS%3Acdrw&acl=world%3Aanyone%3Ar
The scheme ends at the first colon and the permissions start after the last one, so ids containing commas (x509 DNs) or colons (digest) are kept intact. Permissions are any of c, d, r, w, a, or all. The auth scheme takes an empty id (auth::cdrwa); every other scheme requires one.
Caveats:
The REST server's own ZooKeeper identity must be granted CREATE by the ACL (and DELETE when recreate=true). Otherwise creation fails with a 500 and the partial state described in Support custom ZooKeeper ACLs when creating a cluster #224 can remain.
WRITE is only needed for the Azure cloud provider, which also updates the cluster config. The cloud config node itself is created with the client default ACL.
auth::perms expands to the authenticated identities of the REST server's ZooKeeper session. If that session is unauthenticated, ZooKeeper rejects it as an invalid ACL and the request returns 500.
The x509 id format depends on how the ensemble derives client ids from certificates. Stock ZooKeeper uses the subject DN; other deployments may derive a different id.
Digest ACLs cannot be used for the REST server's own session because RealmAwareZkClient has no addAuthInfo. Use x509 or SASL.
Ensembles that assign ACLs server side (setX509ClientIdAsAcl) ignore the supplied list, and ACLs have no effect with skipACL=yes.
The ClusterSetup command line is unchanged.
Request acl values are caller-controlled, so who may set them is governed by the existing authorization on cluster creation.
Tests
The following tests are written for this issue:
TestClusterAcl (helix-rest, embedded ZooKeeper): request acl parameters applied to the cluster root, metadata nodes and every built-in state model definition; multiple entries keep their order; read merged into a requested world:anyone entry; malformed entries return 400 and create nothing; no acl keeps the default; auth:: without an authenticated session returns 500 and creates nothing; a digest ACL that excludes the REST server returns 500.
TestClusterAclParser (helix-rest): permissions, ids with commas and colons, auth:: accepted and empty ids rejected for other schemes, unmodifiable result, malformed entries with the format hint, world-read append and merge.
TestClusterSetupWithAcl (helix-core): ACL passed through ClusterSetup to the root and every built-in state model definition, also on recreate; the ACL-aware addStateModelDef respects recreateIfExists, and an invalid or null definition is rejected before anything is deleted; null ACL keeps the default.
The following is the result of the "mvn test" command on the appropriate module:
Run locally on JDK 11 with mvn -o -pl helix-rest -am test -Dtest=... -Dcheckstyle.skip. All of these passed, with no failures or errors:
The full helix-rest and helix-core suites were not run locally.
Also checked by hand earlier against a running HelixRestMain and a local ZooKeeper: curl -G -X PUT .../admin/v2/clusters/X --data-urlencode "acl=world:anyone:r" --data-urlencode "acl=ip:127.0.0.1:cdrwa" returned 201 and getAcl on the cluster znodes matched; acl=not-an-acl returned 400; no acl kept world:anyone:cdrwa. x509 ACLs were not exercised because that ZooKeeper has no x509 auth provider.
Local code review completed
Changes that Break Backward Compatibility (Optional)
None. The existing ClusterSetup#addCluster overloads keep their behavior; the new ACL path is only used when a request acl parameter is present. ClusterAccessor#createCluster gains one parameter, which only matters to callers invoking it directly rather than over HTTP. HelixAdmin#addStateModelDef gains a default method, so existing implementations still compile.
Documentation (Optional)
No wiki page added.
Commits
My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"
Some commit subjects are longer than 50 characters. Bodies wrap at 72. The branch has six commits; the third removes an earlier provider and namespace-manifest approach.
Code Quality
My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
The formatter was not run; the code follows the style of the surrounding files.
Clusters created via PUT /clusters/{id} still got the client default ACL,
because ClusterAccessor -> ClusterSetup only called the two-argument
HelixAdmin#addCluster. Wire the ACL-aware overload added in linkedin#224 through
the REST path.
- ClusterSetup: add addCluster(name, overwrite, cloudConfig, List<ACL>);
existing overloads delegate with a null ACL.
- ClusterAclProvider / NoopClusterAclProvider: pluggable per-request ACL
source, passed to HelixRestServer through a new constructor overload.
- HelixRestNamespace: optional CLUSTER_ACL (list of scheme:id:perms),
parsed by ClusterAclParser and read from the namespace manifest. It is
not exposed through the namespaces endpoint.
- ClusterAccessor#createCluster resolves the ACL as provider, then
namespace ACL, then the client default (unchanged behavior).
The REST server's own ZooKeeper identity must be granted CREATE by the
ACL (and WRITE when a cloud config is supplied). As in linkedin#224, only nodes
created by addCluster are covered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PUT /clusters/{id} now takes repeatable acl query parameters, each a
scheme:id:permissions entry. They take precedence over the
ClusterAclProvider and the namespace CLUSTER_ACL, so callers of a
deployed REST server can set the ACL without a manifest or a custom
main. A malformed entry returns 400 before anything is created.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This rejects ZooKeeper's valid auth::cdrwa form because the built-in auth scheme intentionally uses an empty ID. As a result, callers cannot configure an ACL bound to the REST server session's authenticated identities through either query parameters or the manifest. Permit an empty ID for the auth scheme while retaining the empty-ID rejection for other schemes, and add a parser case for it.
This overstates the required permission: addCloudConfig creates /CONFIGS/CLOUD/{cluster}; only the Azure branch below updates the existing cluster-config node. Saying every cloud config requires WRITE can cause operators to overgrant privileges. Limit this requirement to Azure cloud configuration (and make the corresponding ClusterAclProvider wording match).
Clarify Azure-only WRITE requirement for ACL providers
The WRITE requirement is only triggered by the Azure topology update in ClusterSetup; other cloud configurations are written to a newly created cloud-config znode. Keep this public extension-point contract precise so provider implementers do not grant unnecessary write access.
helix-rest is a single shared deployment, so there is no custom launcher
to plug a ClusterAclProvider into and the default namespace cannot be
configured through a manifest. Keep only the acl query parameter on
cluster creation and remove the provider SPI, the namespace CLUSTER_ACL
property and the manifest parsing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When a create-cluster request carries acl parameters, add world:anyone
read to the ACL, merging it into a requested world:anyone entry. A
request without acl keeps the client default, which already allows
read. Malformed acl entries now return the offending entry and the
expected scheme:id:permissions format with examples, and the endpoint
Javadoc documents the parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
This loop asserts the ACL on the /STATEMODELDEFSparent but never on its child definition records (/STATEMODELDEFS/MasterSlave, /LeaderStandby, …). Those children aren't created by _admin.addCluster(...) / createZKPaths — they're written afterward by ClusterSetup.addCluster's addStateModelDef(...) loop, which goes through the data accessor with no ACL and therefore lands at the ZkClient default OPEN_ACL_UNSAFE (world:cdrwa — world-writable and world-ADMIN).
Because this assertion only checks the parent, the test passes while the behavior-governing state-model definitions remain world-writable. This masks the gap already flagged on ClusterSetup.java. Recommend asserting the ACL on at least one concrete definition, e.g. PropertyPathBuilder.stateModelDef(cluster) + "/MasterSlave", so the hole is caught here.
The reason will be displayed to describe this comment to others. Learn more.
Fixed in d135f99. assertClusterAcl in TestClusterAcl and the helix-core TestClusterSetupWithAcl now assert the ACL on each built-in state model definition node, not only the /STATEMODELDEFS parent. This caught the real gap: those children previously kept the open default ACL. It is fixed in ClusterSetup/ZKHelixAdmin as described in the first thread. I also dropped the ip:127.0.0.1-only test case that failed on CI, because the runner's ZooKeeper session was not matched by that IP. The remaining integration cases include world:anyone so the test client can create and drop the cluster. Appending read to a non-world ACL is covered by TestClusterAclParser.
The reason will be displayed to describe this comment to others. Learn more.
withWorldRead unconditionally grants world:anyone READ — merged into an existing world entry here, or appended when none is present. For the default (no-acl) path this is harmless since the client default is already OPEN_ACL_UNSAFE. But when a caller supplies a deliberately restrictive ACL — say only x509:myservice:cdrwa — the resulting cluster metadata is still world-readable with no way to opt out. In other words, this endpoint can never create a cluster whose metadata is private to specific identities.
If that's intentional (e.g. Helix clients/controllers connecting under varied identities need broad read), that's defensible — but it's worth confirming it satisfies tenant security requirements, and considering whether read should instead be granted to a specific reader identity rather than world:anyone, or exposed as an opt-out.
The reason will be displayed to describe this comment to others. Learn more.
Keeping this as is. Always granting world:anyone read is intentional. Helix participants, controllers and spectators connect under varied identities and need to read cluster metadata. Without an acl parameter the default ACL already grants world read, so this keeps the two paths consistent. Callers still control write, delete and admin access. If an opt-out is needed later it can be added as an explicit parameter, but I'd prefer not to add one without a concrete use case.
The reason will be displayed to describe this comment to others. Learn more.
The parser validates the format (non-empty scheme/id/perms, valid perm chars) but never the scheme itself. An unknown or malformed scheme such as foo:bar:r passes parsing cleanly, then fails server-side at setACL with InvalidACL — surfacing as a 500 with partial, un-rolled-back cluster state (cf. testServerOutsideAclFailsCreation) instead of a clean 400.
Since the goal here is to reject bad input before anything is created, consider validating scheme against the schemes the ensemble actually supports (world/auth/digest/ip/x509/sasl) up front, so a typo'd scheme is caught at the 400 boundary rather than leaving a half-created cluster behind.
The reason will be displayed to describe this comment to others. Learn more.
Keeping this as is. A scheme whitelist would reject ensemble-specific providers (x509, sasl, custom ones), so it would need to be configured per ensemble. An unsupported scheme or malformed id fails on the first node create with InvalidACL. That returns 500 and leaves no partial cluster; I confirmed this locally with x509 on an ensemble without that provider. Format errors the parser can detect, such as a bad shape, missing id or bad permission letters, are rejected up front with 400 and the expected format in the message.
…iew feedback
- Create built-in state model definition nodes with the requested ACL. ZooKeeper does not
inherit ACLs, so they previously kept the open default ACL. Adds an ACL-aware
HelixAdmin#addStateModelDef overload (default throws for non-empty ACLs, ZKHelixAdmin
implements it) used by ClusterSetup#addCluster.
- Accept the auth scheme with an empty id (auth::perms); other schemes still require an id.
- Use a generic x509 DN in the ACL format hint and Javadoc.
- Document that WRITE is only needed for the Azure cloud config, and that the cloud config
node keeps the default ACL.
- Make the REST ACL integration tests independent of the runner's IP, assert ACLs on every
built-in state model definition, and cover auth without an authenticated session.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addressing the review note on cloud config in d135f99: the ClusterSetup#addCluster Javadoc and PR description no longer say a cloud config always needs WRITE. addCloudConfig only creates the cloud config node, which uses the client default ACL. WRITE is needed only for the Azure provider, which also updates the cluster config.
This security boundary is inaccurate when addCloudConfig=true: ClusterSetup creates the cloud-config znode through the default-ACL path, not with the requested ACL. Document that exception here so REST callers do not assume all nodes created by this request are protected.
The reason will be displayed to describe this comment to others. Learn more.
recreateIfExists deletes the existing definition before this validation runs.
When recreateIfExists == true and a non-empty acl is supplied, the node at stateModelPath has already been removed by _zkClient.deleteRecursively(stateModelPath) in the exists/recreateIfExists block above by the time we reach this stateModel == null || !stateModel.isValid() check. A null or invalid model therefore destroys the existing (valid) definition and then throws, leaving the cluster with no definition at that path.
This isn't reachable through the REST path (built-in defs are always valid), but it's a real data-loss ordering bug for any direct caller of this ACL-aware overload. Suggest validating before the destructive delete — e.g. hoist the null/validity check above the exists block, while keeping the early return for the recreateIfExists == false case.
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 9f01e87. For a non-empty ACL, the null/validity check now runs before the exists/recreateIfExists block, so a rejected call no longer deletes the existing definition. The early return for recreateIfExists == false is unchanged, and so is the null/empty-ACL path. Added testInvalidStateModelDefWithAclDoesNotDeleteExistingDefinition: a recreate with an invalid or null model now throws and leaves the existing definition and its ACL intact. I confirmed that test fails without the fix.
With a non-empty ACL, addStateModelDef deleted the existing definition
when recreateIfExists was true and only then rejected a null or invalid
model, leaving the cluster without that definition. Validate first so a
rejected call leaves the existing definition and its ACL untouched.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For an entry such as ip:127.0.0.1:rx, parsePermissions reports only Invalid ACL permissions "rx". The HTTP 400 response therefore omits the offending entry, unlike structural-validation errors and the documented behavior. With repeatable acl parameters, this makes the invalid entry harder to locate. Include the complete entry when propagating permission-validation errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues
Follow-up to #224 (see the review thread on
HelixAdmin.javaasking aboutClusterSetupand the RESTClusterAccessor). There is no tracking issue.Description
#224 added
HelixAdmin#addCluster(String, boolean, List<ACL>), but clusters created through helix-rest (PUT /clusters/{id}) still got the client default ACL.ClusterAccessorcallsClusterSetup, which only called the two-argumentaddCluster. This PR lets the caller of cluster creation pass an ACL through that path. helix-rest is a single shared deployment, so the ACL is supplied per request rather than through a custom launcher or a namespace manifest.ClusterSetup: newaddCluster(name, overwrite, cloudConfig, List<ACL>). The existing overloads delegate with anullACL, so current callers are unchanged. The ACL is applied to the cluster root, the metadata nodes and the built-in state model definitions.HelixAdmin#addStateModelDef(..., boolean recreateIfExists, List<ACL>): new overload. The default implementation delegates when the ACL is null or empty and throwsUnsupportedOperationExceptionotherwise;ZKHelixAdmincreates the definition node with the ACL. The existing overloads are unchanged.ClusterAclParser(org.apache.helix.rest.acl): parsesscheme:id:permissionsentries intoACLs. Errors name the offending entry and show the expected format with examples.PUT /clusters/{id}accepts repeatable, URL-encodedaclquery parameters, documented in the method Javadoc. A malformed entry returns 400 before anything is created.world:anyoneis always granted read in addition to the requested ACL (merged into a requestedworld:anyoneentry). Withoutacl, the client default ACL is kept, which already allows read.Example:
The scheme ends at the first colon and the permissions start after the last one, so ids containing commas (x509 DNs) or colons (digest) are kept intact. Permissions are any of
c,d,r,w,a, orall. Theauthscheme takes an empty id (auth::cdrwa); every other scheme requires one.Caveats:
CREATEby the ACL (andDELETEwhenrecreate=true). Otherwise creation fails with a 500 and the partial state described in Support custom ZooKeeper ACLs when creating a cluster #224 can remain.WRITEis only needed for the Azure cloud provider, which also updates the cluster config. The cloud config node itself is created with the client default ACL.auth::permsexpands to the authenticated identities of the REST server's ZooKeeper session. If that session is unauthenticated, ZooKeeper rejects it as an invalid ACL and the request returns 500.x509id format depends on how the ensemble derives client ids from certificates. Stock ZooKeeper uses the subject DN; other deployments may derive a different id.RealmAwareZkClienthas noaddAuthInfo. Use x509 or SASL.addClusterand the built-in state model definitions carry the ACL. Resources and instances written later use the default ACL.setX509ClientIdAsAcl) ignore the supplied list, and ACLs have no effect withskipACL=yes.ClusterSetupcommand line is unchanged.aclvalues are caller-controlled, so who may set them is governed by the existing authorization on cluster creation.Tests
The following tests are written for this issue:
TestClusterAcl(helix-rest, embedded ZooKeeper): requestaclparameters applied to the cluster root, metadata nodes and every built-in state model definition; multiple entries keep their order; read merged into a requestedworld:anyoneentry; malformed entries return 400 and create nothing; noaclkeeps the default;auth::without an authenticated session returns 500 and creates nothing; a digest ACL that excludes the REST server returns 500.TestClusterAclParser(helix-rest): permissions, ids with commas and colons,auth::accepted and empty ids rejected for other schemes, unmodifiable result, malformed entries with the format hint, world-read append and merge.TestClusterSetupWithAcl(helix-core): ACL passed throughClusterSetupto the root and every built-in state model definition, also on recreate; the ACL-awareaddStateModelDefrespectsrecreateIfExists, and an invalid or null definition is rejected before anything is deleted; null ACL keeps the default.The following is the result of the "mvn test" command on the appropriate module:
Run locally on JDK 11 with
mvn -o -pl helix-rest -am test -Dtest=... -Dcheckstyle.skip. All of these passed, with no failures or errors:TestClusterAcl,TestClusterAclParser,TestClusterAccessor,TestHelixRestServerTestClusterSetupWithAcl,TestClusterSetup,TestZkHelixAdminThe full helix-rest and helix-core suites were not run locally.
Also checked by hand earlier against a running
HelixRestMainand a local ZooKeeper:curl -G -X PUT .../admin/v2/clusters/X --data-urlencode "acl=world:anyone:r" --data-urlencode "acl=ip:127.0.0.1:cdrwa"returned 201 andgetAclon the cluster znodes matched;acl=not-an-aclreturned 400; noaclkeptworld:anyone:cdrwa. x509 ACLs were not exercised because that ZooKeeper has no x509 auth provider.Changes that Break Backward Compatibility (Optional)
ClusterSetup#addClusteroverloads keep their behavior; the new ACL path is only used when a requestaclparameter is present.ClusterAccessor#createClustergains one parameter, which only matters to callers invoking it directly rather than over HTTP.HelixAdmin#addStateModelDefgains a default method, so existing implementations still compile.Documentation (Optional)
Commits
Some commit subjects are longer than 50 characters. Bodies wrap at 72. The branch has six commits; the third removes an earlier provider and namespace-manifest approach.
Code Quality
(helix-style-intellij.xml if IntelliJ IDE is used)
The formatter was not run; the code follows the style of the surrounding files.
🤖 Generated with GitHub Copilot CLI