Strategy ONE

Migrate from ZooKeeper to KRaft Mode

Starting in Strategy One (December 2025), Kafka will operate exclusively in KRaft mode in Platform Analytics. The update removes the requirement and availability of the ZooKeeper service to enhance performance, improve resilience, and align with the latest Kafka architecture, as of version 4.10.

Be aware that migrating existing data is NOT supported. All existing telemetry data must be fully processed before upgrading to Strategy One (December 2025) or later.

Key Upgrade Impacts

  1. Data Migration Limitations: No Historical Data Migration

    Kafka v4.1.0 and later do not support migrating historical data or consumer group offsets from ZooKeeper to KRaft mode.

    • Existing data in your current Kafka cluster (with ZooKeeper) will NOT be available post-upgrade.

    • Customers are required to process all existing telemetry data prior to initiating the upgrade. This is necessary to avoid data loss, as unprocessed records and offsets cannot be transferred or preserved.

    You must complete the following before upgrading:

    • Process all in-flight data: Ensure that all Platform Analytics consumers have finished consuming all topic partitions and clear your backlogs.

    • Validate consumer completion: Review the lu_topic_partition_offset table to verify that offsets no longer increase, which indicates the consumer has finished processing data from all topics.

    • For on-premises and instance-based users, if you inadvertently upgrade without processing all data, you can temporarily restart the cluster in the old ZooKeeper mode to process remaining records before the final migration to KRaft.

      If you upgrade without processing all data, any unprocessed telemetry will be permanently lost.

  2. Architectural Changes: Elimination of ZooKeeper

    ZooKeeper is no longer included in the Kafka deployment starting in v4.1.0.

    • ZooKeeper service or containers/pods are no longer required.

    • All cluster metadata and coordination occurs within Kafka via the Raft consensus protocol (KRaft).

  3. Upgrade Requirements

    • All brokers must upgrade at once: Mixed-mode clusters are not supported. You cannot run some brokers in ZooKeeper and others in KRaft simultaneously.

    • Network configuration: Open required ports for controller-quorum traffic and update the firewall/network policies, as needed.

Configure a 3-Node Kafka Cluster in KRaft Mode for Platform Analytics

A single node Kafka deployment is available out-of-the-box by the Windows and Linux installer. Use the following steps to deploy a 3 node cluster.

Prerequisites

Nodes: You must have three Linux servers:

  • Node 1: <IP Node1>

  • Node 2: <IP Node2>

  • Node 3: <IP Node3>

Cluster ID: Use a universally unique identifier (UUID); for example, 11111111-1111-1111-1111-111111111111.

Configure the 3-Node Kafka Cluster

  1. Create or edit the config/server.properties file on each node, as follows:

    • Node 1 (<IP Node1>):

      Copy 
      node.id=1
      listeners=PLAINTEXT://<IP Node1>:9092,CONTROLLER://<IP Node1>:9093
      advertised.listeners=PLAINTEXT://<IP Node1>:9092
      controller.quorum.bootstrap.servers=<IP Node1>:9093,<IP Node2>:9093,<IP Node3>:9093
      controller.quorum.voters=1@<IP Node1>:9093,2@<IP Node2>:9093,3@<IP Node3>:9093
      offsets.topic.replication.factor=3
      share.coordinator.state.topic.replication.factor=3
      transaction.state.log.replication.factor=3

    • Node 2 (<IP Node2>):

      Copy 
      node.id=2
      listeners=PLAINTEXT://<IP Node2>:9092,CONTROLLER://<IP Node2>:9093
      advertised.listeners=PLAINTEXT://<IP Node2>:9092
      controller.quorum.bootstrap.servers=<IP Node1>:9093,<IP Node2>:9093,<IP Node3>:9093
      controller.quorum.voters=1@<IP Node1>:9093,2@<IP Node2>:9093,3@<IP Node3>:9093
      offsets.topic.replication.factor=3
      share.coordinator.state.topic.replication.factor=3
      transaction.state.log.replication.factor=3

    • Node 3 (<IP Node3>):

      Copy 
      node.id=3
      listeners=PLAINTEXT://<IP Node3>:9092,CONTROLLER://<IP Node3>:9093
      advertised.listeners=PLAINTEXT://<IP Node3>:9092
      controller.quorum.bootstrap.servers=<IP Node1>:9093,<IP Node2>:9093,<IP Node3>:9093
      controller.quorum.voters=1@<IP Node1>:9093,2@<IP Node2>:9093,3@<IP Node3>:9093
      offsets.topic.replication.factor=3
      share.coordinator.state.topic.replication.factor=3
      transaction.state.log.replication.factor=3

  2. Run the following on each node using the same cluster ID on all nodes:

    Copy 
    bin/kafka-storage.sh format -t 11111111-1111-1111-1111-111111111111 -c config/server.properties

  3. On each node use the following to start Kafka as a daemon:

    Copy 
    bin/kafka-server-start.sh -daemon config/server.properties

  4. On any node, check the KRaft quorum status:

    Copy 
    bin/kafka-metadata-quorum.sh --bootstrap-server <IP Node3>:9092 describe --status

    Details similar to the following should display:

    Copy 
    ClusterId:              11111111-1111-1111-1111-111111111111
    LeaderId:               3
    CurrentVoters: [{"id": 1, "endpoints": [...]}, {"id": 2, ...}, {"id": 3, ...}]
    ...

  5. Configure Platform Analytics and the Intelligence Server by using the following:

    • Platform Analytics Consumer (PAConsumerConfig.yaml):

      Copy 
      kafkaTopicNumberOfReplicas: 3
      kafkaConsumerProperties:
          bootstrap.servers: "<IP Node1>:9092,<IP Node2>:9092,<IP Node3>:9092"

    • Intelligence Server (via Command Manager):

      Copy 
      ALTER SERVER CONFIGURATION ENABLEMESSAGINGSERVICES TRUE CONFIGUREMESSAGINGSERVICES "bootstrap.servers:<IP Node1>:9092,<IP Node2>:9092,<IP Node3>:9092/batch.num.messages:5000/queue.buffering.max.ms:2000";

    • Library Config:

      Copy 
      producer.kafkaProperties.bootstrap.servers = <IP Node1>:9092,<IP Node2>:9092,<IP Node3>:9092

  6. Verify that the Kafka topic data folders, kafka_data, are available on each node after starting Kafka.

  7. Send the following test message:

    Copy 
    bin/kafka-console-producer.sh --bootstrap-server <IP Node3>:9092 --topic Mstr.PlatformAnalytics.IsReportSecurityFilterStats

    Consume and test the following message:

    Copy 
    bin/kafka-console-consumer.sh --bootstrap-server <IP Node3>:9092 --topic Mstr.PlatformAnalytics.IsReportSecurityFilterStats --from-beginning

    A message should display, confirming cross-node replication and cluster health.

Additional Information

  • No ZooKeeper service is required - do NOT configure any ZooKeeper properties.

  • All three nodes must use the same cluster ID.

  • The replication factor must equal the number of brokers for fault tolerance.

Frequently Asked Questions

1. Can I migrate my existing Kafka data and offsets into the new cluster?

No. Historical data migration from ZooKeeper to KRaft clusters is not supported. All existing data must be processed before upgrading.

2. Will there be a ZooKeeper service after upgrading?

No. ZooKeeper is fully removed from the architecture. All coordination is managed within Kafka via Raft.

Kafka Service Script Updates

The Linux service script under /etc/init.d is renamed from mstr-user@install_time-kafka-zookeeper to mstr-user@install_time-kafka.

New Kafka Service Commands

The control command has changed from kafka-zookeeper to kafka:

Copy 
service microstrategy kafkastart | kafkastop | kafkastatus

3. What happens if I accidentally upgrade with unprocessed data?

Your data will be lost. For on-premise deployments, you can temporarily start the old Kafka (ZooKeeper mode), finish processing, and reupgrade.