MicroStrategy ONE

Configure High Throughput or Advanced Architecture

This topic explains how to configure high throughput architecture using a cluster of Kafka nodes (Telemetry servers). One Telemetry Store (Platform Analytics) can only consume data from a single Kafka node or single Kakfa cluster.

All Kafka nodes should be in cluster and multiple Kafka clusters are not supported.

Get started with the following topics:

1. Install Components

2. Configure Telemetry Server

3. Restart Necessary Services

4. Configure Platform Analytics Consumer

1. Install Components

Start by installing components on the corresponding environments.

  1. Choose the following components for machines 1, 2, and 3 in the Installation wizard:

    • MicroStrategy Intelligence

    • MicroStrategy Telemetry Server

    • Choose Create a cluster... while creating a clustered environment for Telemetry and provide the other nodes or machines addresses where you have installed or are going to install Telemetry. Repeat this for machines 1, 2, and 3.

  2. Choose the following components for machines 4 and 5 in the Installation wizard:

    • MicroStrategy Intelligence

    • MicroStrategy Telemetry Server

  3. After installation, turn off MicroStrategy Telemetry server (Kafka & Zookeeper services).

  4. Choose the following components for machine 6 in the Installation wizard:

    • Platform Analytics

    • During the installation of components, you must enter repository (machine 7) connection information.

  5. Install Platform Analytics Repository on machine 7. You can choose an out-of-the-box MicroStrategy Repository or opt for you own instance of PostgreSQL database server, provided by your organization.

  6. For Windows deployments, continue with Windows Specific Modifications to Create the Platform Analytics Service, otherwise go to 2. Configure Telemetry Server for Linux deployments.

Windows Specific Modifications to Create the Platform Analytics Service

On Windows machines, you must recreate the Platform Analytics service to prevent the service from going down.

  1. Go to Services.

    1. Stop MicroStrategy Platform Analytics Consumer.

    2. Stop MicroStrategy Platform Analytics In-Memory Cache.

  2. Delete the MicroStrategy Platform Analytics Consumer service.

    1. Launch a Windows command prompt with administrative privileges.

    2. Execute the following command:

      Copy 
      sc delete MSTR_PlatformAnalyticsConsumer

    3. Close Services.

  3. Recreate the Platform Analytics Consumer service.

    1. Navigate to the Platform Analytics directory.

    2. Open MSTR_PlatformAnalyticsConsumer.config for editing.

    3. Delete --DependesOn =Redis to remove the dependent services (Kafka and Zookeeper).

  4. Launch a Windows command prompt with administrative privileges.

    1. Navigate to the Platform Analytics directory.

    2. Execute the following command:

      Copy 
      PlatformAnalyticsConsumer.exe install MSTR_PlatformAnalyticsConsumer --Config PlatformAnalyticsConsumer_config.txt

  5. Go to Services.

    1. Start MicroStrategy Platform Analytics Consumer.

    2. Refresh Service Manager if necessary.

2. Configure Telemetry Server

Perform following steps below for all Telemetry server nodes. This example uses machines 1, 2, and 3.

Edit server.properties

  1. Open server.properties for editing.

    Windows location:

    C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_x.x.xx\config

    Linux location:

    /opt/MicroStrategy/MessagingServices/Kafka/kafka_x.x.x./config

  2. Under ##### Server Basics ####, provide a unique broker ID to each Telemetry server machine in the preferred order of node failover.

    In this example:

    Machine 1: broker.id=1
    Machine 2: broker.id=2
    Machine 3: broker.id=3

    Copy 
    # Set the broker id to a unique value for each node. 
    # Do not change it on the machine configured during single node set up, i.e. your main node. It should be left at the default value and referred to by the other nodes.
    # For example,

    broker.id=1

  3. Under ##### Internal Topic Settings ####, set both offsets and transaction state factors to the amount of nodes in the cluster. In this example, that is 3.

    Copy 
    # offsets.topic.replication.factor= set to the number of nodes in your cluster
    # transaction.state.log.replication.factor= set to the number of nodes in your cluster
    # For example,

    offsets.topic.replication.factor=3
    transaction.state.log.replication.factor=3

  4. Under ##### Zookeeper #####, add all Telemetry server node IP addresses or FQDNs for the zookeeper.connect parameter. The order of nodes must correspond to the broker ID parameter in step 2.

    Copy 
    # Set zookeeper.connect= to a comma separated list of <IP address:2181> for all nodes in the cluster.
    # For example, 

    zookeeper.connect=10.27.18.73:2181,10.27.18.224:2181,10.27.36.168:2181

Edit zookeeper.properties

  1. Open zookeeper.properties for editing.

    Windows location:

    C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_x.x.xx\config

    Linux location:

    /opt/MicroStrategy/MessagingServices/Kafka/kafka_x.x.x./config

  2. Add new lines at the end of the file with server.node_id=ip:2888:3888. In this example, there are three new lines for each node.

    Copy 
    # To allow Zookeeper to work with the other nodes in your cluster, add the following properties to the end of the zookeeper.properties file.
    # initLimit=5
    # syncLimit=2
    # server.X= <IP address of the node>:2888:3888
    # When adding this property, replace X above with the broker.id for the node being referenced. A separate entry must be made for each node in the cluster.
    # For example,

    initLimit=5
    syncLimit=2
    server.0=10.27.18.73:2888:3888
    server.1=10.27.18.224:2888:3888
    server.2=10.27.36.168:2888:3888

Edit myid

  1. Open myid for editing. If this file doesn’t exist, you should create one.

    Windows location:

    C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\zookeeper

    Linux location:

    /opt/MicroStrategy/MessagingServices/tmp/zookeeper

  2. Confirm that the myid file does not have a hidden extension. In File Explorer, go to View > Show > File name extensions to show extensions. If your file has an extension, remove it.

  3. Make sure the broker.id for each node matches the values you set in server.properties.

    Copy 
    # Make sure the broker.id is the same as it appears in server.properties.
    # For example,

    broker.id=1

3. Restart Necessary Services

After updating the configurations for Kafka and Zookeeper on all nodes in the cluster, you must restart the services, including the Intelligence server.

When restarting the services, it's important to note that all configuration file changes must be completed first. For example, if you are adding two additional Kafka nodes and you already have one existing node, then the install and configuration should be completed on all three nodes before restarting any of the services.

Additionally, some services are dependent on each other, so the services should be started in the order provided below. Not starting in this order can cause inconsistencies in the services.

  1. Start Zookeeper and Kafka on the main node before starting other nodes.

  2. Start Zookeeper on the remaining nodes.

  3. Start Kafka on the remaining nodes.

4. Configure Platform Analytics Consumer

Perform the following steps on the node where you are running Platform Analytics Consumer. In this example, that is machine 6.

  1. Open PAConsumerConfig.yaml for editing.

    Windows location:

    C:\Program Files (x86)\MicroStrategy\Platform Analytics\conf

    Linux location:

    /opt/MicroStrategy/Platform Analytics/conf

  2. Add all telemetry node IP addresses to the file, using the following format:

    zookeeperConnection:IP1:port,IP2:port,IP3:port

    bootstrap.servers: IP1:port,IP2:port,IP3:port

    Copy 
    # Set kafkaTopicNumberOfReplicas: number of nodes in cluster
    # Set zookeeperConnection: <ipAddress:2181> for all nodes in cluster
    # Set bootstrap.servers: <ipAddress:9092> for all nodes in cluster
    # For example,

    kafkaTopicNumberOfReplicas: 3
    zooKeeperConnection: 10.27.18.73:2181,10.27.18.224:2181
    bootstrap.servers: 10.27.18.73:9092,10.27.18.224:9092

  3. Ensure that the kafkaTopicNumberOfReplicas parameter matches the number of Telemetry server nodes. In this example, that would be 3.

  4. Restart the following services:

    • MicroStrategy Platform Analytics Consumer

    • MicroStrategy Platform Analytics In-Memory Cache