MicroStrategy ONE

Add an Additional Kafka Node to an Existing Kafka Cluster Post Installation

  • Windows
  • Linux

After the initial installation of Platform Analytics is completed, multiple nodes containing Telemetry Server(s) can be added in order to create clustered Telemetry Servers. The clustered environment ensures that if a telemetry node is down, there is a copy of the telemetry log available on another node.

To manually add a new Kafka node to an existing Kafka cluster on a Windows environment, perform the following steps:

  1. Disable Services
  2. Install the Telemetry Server
  3. Configure Kafka
  4. Restart Services

Multiple services were renamed in the MicroStrategy 2019 release. Because this guide requires modifying the underlying files, it uses the original service name.

To add an additional Kafka node, you need:

  • One environment with MicroStrategy and Platform Analytics fully installed and configured. For more information, see Installing Platform Analytics.
  • Idle environment(s) to be added as the Kafka Server node(s) to create a Kafka cluster.

Disable Services

Before configuring the new Kafka nodes, ensure that the Intelligence Server Producer, Apache ZooKepper, Apache Kafka, and Platform Analytics Consumer and Producer are disabled. If you already have more than one node in the cluster, disable the services on all nodes.

  1. In Command Manager, disable the Intelligence Server Producer by running:
    Copy
    ALTER SERVER CONFIGURATION ENABLEMESSAGINGSERVICES FALSE;
  2. Open Windows Services and locate Apache ZooKeeper, Apache Kafka, and MicroStrategy Platform Analytics Consumer and MicroStrategy Usher Metadata Producer. Right-click each service and select Stop.

Install the Telemetry Server

  1. Open the MicroStrategy Installation Wizard on the node(s) you want to add additional Kafka Servers to.
  2. Step through the Wizard.
  3. When you're prompted to select the features you want to install, select Telemetry Server.

    MicroStrategy Command Manager is installed by default.

Configure Kafka

Perform following steps for all nodes, including those that already exist in the cluster.

  1. Open the server.properties file located in the C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-1.1.0\config directory.
  2. Modify the file:
    1. Set the broker.id to a unique value for each node.

      Do not change the broker.id on your main node (the machine configured during single node set up). It should remain at the default value and be referred to by the other nodes.

    2. Set offsets.topic.replication.factor= and transaction.state.log.replication.factor= to the number of nodes in your cluster.
    3. Set zookeeper.connect= to a comma separated list of <IP address:2181> for all nodes in the cluster.
    4. Add advertised.listeners=<the IP address for this node> at the end of the file.
  3. Save the file.
  4. Open the zookeeper.properties file found in the same directory.
  5. Add the following properties to the end of the zookeeper.properties file:
    • initLimit=5
    • syncLimit=2

    • server.X= <IP address of the node>:2888:3888

      Replace X with the broker.id of the node being referenced. A separate entry must be made for each node in the cluster.

      For example:

      Copy
      initLimit=5
      syncLimit=2
      server.0=10.27.18.73:2888:3888
      server.1=10.27.18.224:2888:3888

  6. Create a text file called myid containing only the broker.id of the node.

    If your broker.id=1, enter 1.

  7. Save the file in the ZooKeeper directory located at C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\zookeeper.

    Ensure the file doesn't have a hidden extension. To check, click Check View > Show/hide > File name extensions in File Explorer. Delete any extension of your myid file.

Restart Services

After the installation and configuration on all Kafka nodes in the cluster are complete, restart the Intelligence Server Producer, Apache ZooKeeper, Apache Kafka, and Platform Analytics Consumer and Producer.

Before restarting the services, all configuration file changes must be completed first. For example, if you are adding two additional Kafka nodes in addition to your one existing node, then the install and configuration should be completed on all three nodes before restarting any of the services.

Additionally, some services are dependent on each other, so the services must be started in the following order:

Apache Zookeeper

  1. In Windows Services, start Apache ZooKeeper. Start the main node before starting other nodes.

Apache Kafka

  1. In Windows Services, start Apache Kafka.

Intelligence Server Producer

  1. Open Command Manager and run the following script:

    Copy 
    ALTER SERVER CONFIGURATION 
    ENABLEMESSAGINGSERVICES TRUE 
    CONFIGUREMESSAGINGSERVICES"bootstrap.servers:<hostname1:9092>,<hostname2:9092>,<hostname3:9092>/batch.num.messages:5000/queue.buffering.max.ms:2000";

    Replace the hostname and port with the new Telemetry Server cluster configuration for the Platform Analytics environment.

  2. Restart the Intelligence Server.

    If there is a cluster of Intelligence Servers, all nodes must be restarted.

Platform Analytics Consumer

Only perform the following steps on your main node. The main node is the node running Platform Analytics Consumer.

  1. Open the PAConsumerConfig.yaml file located in C:\Program Files (x86)\MicroStrategy\Platform Analytics\conf.
  2. Modify the file:
    1. Set kafkaTopicNumberOfReplicas: to the number of nodes in the cluster.
    2. Set zookeeperConnection: <ipAddress:2181> for all nodes in the cluster.
    3. Set bootstrap.servers: <ipAddress:9092> for all nodes in the cluster.
  3. Save the file.
  4. In Windows Services, start MicroStrategy Platform Analytics Consumer and MicroStrategy Usher Metadata Producer.

Troubleshooting

If Apache ZooKeeper cannot be restarted, ensure Kafka is fully configured.

  1. Open the kafka-logs folder located in C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp.
  2. Open the meta.properties file and ensure the broker.id is the same as it appears in server.properties. If they are different, this may be why Apache ZooKeeper is not starting.
  3. If there is no telemetry in the Kafka topics, check if statistics are enabled for Platform Analytics projects by running the following command in Command Manager:
    Copy
    LIST ALL PROPERTIES FOR PASTATISTICS IN PROJECT "Platform Analytics";

  4. If the command returns False, run:

    Copy 
    ALTER PASTATISTICS BASICSTATS ENABLED DETAILEDREPJOBS TRUE DETAILEDDOCJOBS TRUE JOBSQL TRUE COLUMNSTABLES TRUE IN PROJECT "Platform Analytics";

    BASICSTATS must always be enabled. Select what kind of advanced stats are needed by changing the parameter after it to true/false. See List Platform Analytics Statistics Properties Statement for more information about basic versus advanced statistics.

After the initial installation of Platform Analytics is completed, multiple nodes containing Telemetry Server(s) can be added in order to create clustered Telemetry Servers. The clustered environment ensures that if a telemetry node is down, there is a copy of the telemetry log available on another node.

To manually add a new Kafka node to an existing Kafka cluster on a Linux environment, perform the following steps:

  1. Disable Services
  2. Install the Telemetry Server
  3. Configure Kafka
  4. Restart Services

Multiple services were renamed in the MicroStrategy 2019 release. Because this guide requires modifying the underlying files, it uses the original service name.

To add an additional Kafka node, you need:

  • One environment with MicroStrategy and Platform Analytics fully installed and configured. For more information, see Installing Platform Analytics.
  • Idle environment(s) to be added as the Kafka Server node(s) to create a Kafka cluster.

Disable Services

Before configuring the new Kafka nodes, ensure the Intelligence Server Producer, Apache ZooKepper, Apache Kafka, and Platform Analytics Consumer and Producer are disabled. If you already have more than one node in the cluster, disable the services on all nodes.

  1. In Command Manager, disable the Intelligence Server Producer by running:

    Copy 
    ALTER SERVER CONFIGURATION ENABLEMESSAGINGSERVICES FALSE;

    This operation does not impact standard MicroStrategy Intelligence Server functionality.

  2. In the Platform Analytics directory, located in /opt/MicroStrategy/PlatformAnalytics, open the bin folder.
  3. Run the following commands:
    Copy
    ./platform-analytics-consumer.sh stop
    ./platform-analytics-usher-lookup-producer.sh stop
  4. In the Kafka directory, located in /opt/MicroStrategy/MessagingServices/Kafka/kafka_2.11-1.1.0/, open the bin folder.
  5. Run the following commands:
    Copy
    ./kafka-server-stop.sh
    ./zookeeper-server-stop.sh

Install the Telemetry Server

  1. Open the MicroStrategy Installation Wizard on the node(s) you want to add additional Kafka Servers to.
  2. Step through the Wizard.
  3. When you're prompted to select the features you want to install, select Telemetry Server.

    MicroStrategy Command Manager is installed by default.

Configure Kafka

Perform following steps for all nodes, including those that already exist in the cluster.

  1. Open the server.properties file located in /opt/MicroStrategy/MessagingServices/Kafka/kafka_2.11-1.1.0/config.
  2. Modify the file:
    1. Set the broker.id to a unique value for each node.

      Do not change the broker.id on your main node (the machine configured during single node set up). It should remain at the default value and be referred to by the other nodes.

    2. Set offsets.topic.replication.factor= and transaction.state.log.replication.factor= to the number of nodes in your cluster.
    3. Set zookeeper.connect= to a comma separated list of <IP address:2181> for all nodes in the cluster.
    4. Add advertised.host.name=<the IP address for this node> at the end of the file.
  3. Save the file.
  4. Open the zookeeper.properties file found in the same directory.
  5. Add the following properties to the end of the zookeeper.properties file:
    • initLimit=5
    • syncLimit=2

    • server.X= <IP address of the node>:2888:3888

      Replace X with the broker.id of the node being referenced. A separate entry must be made for each node in the cluster.

      For example:

      Copy
      initLimit=5
      syncLimit=2
      server.0=10.27.18.73:2888:3888
      server.1=10.27.18.224:2888:3888
  6. Create a text file called myid containing only the broker.id of the node.

    If your broker.id=1, enter 1.

  7. Save the file in the ZooKeeper directory located in /opt/MicroStrategy/MessagingServices/Kafka/tmp/zookeeper.

Restart Services

After the installation and configuration on all Kafka nodes in the cluster are complete, restart the Intelligence Server Producer, Apache ZooKepper, Apache Kafka, and Platform Analytics Consumer and Producer.

When restarting the services, it's important to note that all configuration file changes must be completed first. For example, if you are adding two additional Kafka nodes, plus have one existing node, then the install and configuration should be completed on all three nodes before restarting any of the services.

Additionally, some services are dependent on each other, so the services should be started in the following order:

Apache Zookeeper

  1. In the Kafka directory, found in /opt/MicroStrategy/MessagingServices/Kafka/kafka_2.11-1.1.0/, open the bin folder.
  2. Start ZooKeeper on all nodes by running:
    Copy
    ./zookeeper-server-start.sh -daemon ../config/zookeeper.properties

Apache Kafka

  1. In the same folder, start Kafka on all nodes by running:
    Copy
    ./kafka-server-start.sh -daemon ../config/server.properties

Intelligence Server Producer

  1. Open Command Manager and run the following script:
    Copy
    ALTER SERVER CONFIGURATION ENABLEMESSAGINGSERVICES TRUE CONFIGUREMESSAGINGSERVICES "bootstrap.servers:10.27.16.225:9092,10.27.19.34:9092/batch.num.messages:5000/queue.buffering.max.ms:2000";

    Replace the hostname and port with the new Telemetry Server cluster configuration for the Platform Analytics environment.

  2. Restart the Intelligence Server.

    If there is a cluster of Intelligence Servers, all nodes must be restarted.

Platform Analytics Consumer

Only perform the following steps on your main node. The main node is the node running Platform Analytics Consumer.

  1. Open the PAConsumerConfig.yaml file located in the /opt/MicroStrategy/PlatformAnalytics/conf directory.
  2. Modify the file:
    1. Set kafkaTopicNumberOfReplicas: to the number of nodes in the cluster.
    2. Set zookeeperConnection: <ipAddress:2181> for all nodes in the cluster.
    3. Set bootstrap.servers: <ipAddress:9092> for all nodes in the cluster.
  3. Save the file.
  4. In the Platform Analytics directory, located in /opt/MicroStrategy/PlatformAnalytics, open the bin folder.
  5. Run the command:
    Copy
    ./platform-analytics-consumer.sh start
    ./platform-analytics-usher-lookup-producer.sh start

Troubleshooting

If Apache ZooKeeper cannot be started, ensure Kafka is fully configured.

  1. Open the kafka-logs folder located in /opt/MicroStrategy/MessagingServices/Kafka/tmp.
  2. Open the meta.properties file and ensure the broker.id is the same as it appears in server.properties. If they are different, this may be why Apache ZooKeeper is not starting.
  3. If there is no telemetry in the Kafka topics, check if statistics are enabled for Platform Analytics projects by running the following command in Command Manager:
    Copy
    LIST ALL PROPERTIES FOR PASTATISTICS IN PROJECT "Platform Analytics";

  4. If the command returns False, run:

    Copy 
    ALTER PASTATISTICS BASICSTATS ENABLED DETAILEDREPJOBS TRUE DETAILEDDOCJOBS TRUE JOBSQL TRUE COLUMNSTABLES TRUE IN PROJECT "Platform Analytics";

    BASICSTATS must always be enabled. Select what kind of advanced stats are needed by changing the parameter after it to true/false. See List Platform Analytics Statistics Properties Statement for more information about basic versus advanced statistics.