Feature Request to support Kraft based (zookeeper-less) Kafka infrastructure in Apache Storm

[deveshgarg150]

We are using multiple Apache projects including prominent ones like kafka and storm in our org, thanks to ASF for that!

We want to upgrade to Latest LTS of OpenJDK 21 (currently on 1.8) in all our projects which are tightly coupled with each other, as we use to publish internal maven dependencies to be used in our other projects. Now, as of mid-2025, Apache Kafka 4.0.0 is available which has discontinued the use of zookeeper and uses Kraft mode only.

But the latest Apache Storm 2.8.1 is still reliant on zookeeper.

Therefore, we wanted to request a new feature in Apache Storm that supports the latest Kraft based kafka infrastructure. Or if we are mistaken fundamentally, please let us know if that is not possible in the near or distant future.

[rzo1]

Hi,

Apache Storm is essentially in maintenance mode. This means that while the project remains functional and stable and releases are made, the addition of new features is largely dependent on community interest and contributions. In other words, major new features — such as native support for Kafka's KRaft mode — would require active involvement and contributions from developers in the community.

To clarify the current compatibility landscape:

• While Kafka 4.0 has dropped ZooKeeper in favor of KRaft mode, Storm itself still requires ZooKeeper for its internal cluster coordination and operation. This requirement is independent of the Kafka version used.

• That said, you can still use the Kafka client libraries with Kafka 4.0 in your Storm topologies, meaning producing and consuming messages via Storm-Kafka integrations should continue to work, assuming the client versions are compatible.

• However, Storm itself cannot currently run without ZooKeeper. Replacing ZooKeeper with a KRaft-based alternative within Storm would involve (most likely) architectural changes and, as mentioned, would require community interest and contributions to implement. If so, we would need to ensure, that the current ZooKeeper is kept (at least for 2.x) and Storm can work in both modes. Removing ZooKeeper would only be possible in a 3.x version.

That being said, contributions are always welcome, and community-driven development is the primary way forward for feature evolution in Storm at this point.

If this is a critical need for your organization, you might consider opening a discussion on the Storm developer mailing list to get community interest.

Gruß
Richard

[rzo1]

Btw. Storm should Work fine with Java 21 :-)

[deveshgarg150]

Thanks @rzo1 for the detailed reply. Just adding to the context; in the past few days, I've done a POC on local env in the following manner:

• Setup kafka cluster with zookeeper support (Kafka 3.9 series). Used Java 17 for that because otherwise Java 21 gives some Security exceptions while starting the kSQLdb server step.
• Used latest Apache Storm 2.8.1. Setup Storm nimbus, supervisor and ui nodes using Java 21. This nimbus master here was able to employ zookeeper configured in previous step to its own use.
• Shaded the project using Java 21 and submit to storm, works fine though.

[rzo1]

Yep. Kafka's Java level is out of our control :) - might be possible to use a Kafka 4 client (maybe with code updates on Storm side) with Storm. That might be an easier thing to achieve.

[rzo1]

We are now on Kafka 4 on our client side.