Install & Config 2019

MicroStrategy Messaging Services

MicroStrategy Messaging Services on Windows

Messaging Services is a component that is coupled with the Intelligence Server during installations and upgrades. Messaging Services is configured out-of-the-box and runs automatically after the installation is completed.

After installation, you can see the following services are automatically started:

  • Apache Kafka (C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-0.10.1.0)
  • Apache ZooKeeper (C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-0.10.1.0)
  • MicroStrategy Intelligence Server Log Consumer (C:\Program Files (x86)\MicroStrategy\Intelligence Server\KafkaConsumer)

By default MicroStrategy will still send Intelligence Server diagnostic logs to local disk. Diagnostic logs will be sent to the Messaging Services Server after you perform the following:

Afterwards you will see Kafka log files created in the Kafka installation folder:

C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\kafka-logs

Different Kafka topics will be created to store data for different MicroStrategy components.

Configuring Messaging Services after upgrading

By default, MicroStrategy Messaging Services are installed along with the Intelligence server upgrade.

Once you have completed the upgrade process, you need to enable MicroStrategy Messaging Services. If not, the Intelligence Server continues to write to the original log.

Messaging Services Workflow for Intelligence Server

  • Intelligence Server is the Kafka Producer and can be deployed a single node or cluster.
  • Kafka Server can be deployed as a single node or cluster.
  • Intelligence Server Log Consumer can run on any machine that can be connected to a Kafka Server.

Enable MicroStrategy Messaging Services

Messaging Services configuration is saved in the MicroStrategy Intelligence Server configuration. It can be enabled or disabled on the fly, without restarting your Intelligence Server.

Command Manager Scripts for Messaging Services

To check if Messaging Services is enabled, execute:

LIST ALL PROPERTIES FOR SERVER CONFIGURATION;

To enable Messaging Services through Command Manager, execute:

ALTER SERVER CONFIGURATION ENABLEMESSAGINGSERVICES TRUE CONFIGUREMESSAGINGSERVICES "bootstrap.servers:10.15.208.236:9092/batch.num.messages:5000/queue.buffering.max.ms:2000";

In the example above set:

  • bootstrap.servers: to your Kafka Server IP address and port number.
  • batch.num.messages: to the number of messages to send in one batch when using asynchronous mode.
  • queue.buffering.max.ms: to the maximum time to buffer data when using asynchronous mode.

You can specify more Kafka Producer configuration settings in this command following the same format.

Turn On the Sending Log to Messaging Services Feature

You can turn on the Sending Log to Messaging Services feature using either MicroStrategy Web or Command Manager.

From MicroStrategy Web

  1. Log in using and Administrator account.
  2. Open User Preferences > Project Defaults.
  3. Locate Sending Log to Messaging Services in the Features for Customer Feedback section.
  4. Select On from the drop-down menu.
  5. Click Apply.

From Command Manager

  1. Connect to your project source.
  2. Execute the following:

    ALTER FEATURE FLAG "SENDING LOG TO MESSAGING SERVICES" ON;

Modifying Messaging Services Configuration

Apache Kafka Server

The Kafka Server can be configured by modifying the server.properties file found in:

C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-0.10.1.0\config

Both Apache Kafka Server and ZooKeeper should be restarted after modifying the above configuration file.

Intelligence Server Log Consumer

By default the Log Consumer is connecting to the Local Kafka Server.

There are two ways to modify the configuration of Log Consumer:

  1. Delete LogConsumer.properties and execute the following command and follow the steps in the command line:

    C:\Program Files (x86)\MicroStrategy\Intelligence Server\KafkaConsumer>java -jar KafkaConsumer.jar

  2. Modify file C:\Program Files (x86)\MicroStrategy\Intelligence Server\KafkaConsumer\LogConsumer.properties directly.

    The default values after installation are:

    folder_path=C:\\Program Files (x86)\\Common Files\\MicroStrategy\\Log\\DSSErrors # indicate log file location

    is_silent_mode=true # indicate run consumer in silent mode

    broker_port=9092 # Kafka Server port number

    broker_hostname=127.0.0.1 # Kafka Server IP

    poll_time_out=1000 # consumer connection time out limit in seconds

    max_file_size_M=20 # max log file size in MB

    max_num_bak=1 # number of backup files

MicroStrategy Messaging Services Configuration for Clustered Environments

If you have clustered your Intelligence Servers and want to use a separate machine to run MicroStrategy Messaging Services after upgrading, complete the following steps for each node in the cluster.

The minimum number of nodes for a cluster is 3.

Each node must have the following installed:

  • MicroStrategy Messaging Services
  • Apache Kafka
  • Apache Zookeeper

Configure Zookeeper

  1. Browse to folder C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-0.10.1.0\config.
  2. Edit file zookeeper.properties by adding following lines:

    clientPort=2181

    dataDir=C:\\Program Files (x86)\\MicroStrategy\\Messaging Services\\tmp\\zookeeper

    maxClientCnxns=0

    initLimit=5

    syncLimit=2

    server.1=10.27.20.16:2888:3888

    server.2=10.27.20.60:2888:3888

    server.3=10.15.208.236:2888:3888

    Each server parameter must contain a unique integer identifier as shown above. You attribute the server id to each machine by creating a text file named myid, one for each server, which resides in that server's data directory, as specified by the configuration file parameter dataDir = C:\Program Files (x86)\MicroStrategy\Messaging Services\tmp\zookeeper

  3. Go to folder C:\Program Files (x86)\MicroStrategy\MessagingServices\Kafka\kafka_2.11-0.9.0.1\config\zookeeper.

  4. Create a text file named myid containing the identifying value from the server parameter name in the zookeeper.properties file.

Configure Kafka

  1. Browse to folder C:\Program Files (x86)\MicroStrategy\Messaging Services\Kafka\kafka_2.11-0.10.1.0\config.
  2. Edit file server.properties, add a row zookeeper.connect=10.27.20.16:2181,10.27.20.60:2181,10.15.208.236:2181 to the Zookeeper section.

    ############################# Zookeeper #############################

    # Zookeeper connection string (see zookeeper docs for details).

    # This is a comma separated host:port pairs, each corresponding to a zk

    # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

    # You can also append an optional chroot string to the urls to specify the

    # root directory for all kafka znodes.

    # zookeeper.connect=localhost:2181

    zookeeper.connect=10.27.20.16:2181,10.27.20.60:2181,10.15.208.236:2181

  3. Modify the broker.id value to a unique integer from other Kafka servers (the default value is 0), such as for node 10.27.20.60 we use number 2.

    ############################# Server Basics #############################

    # The id of the broker. This must be set to a unique integer for each broker.

    broker.id=2

Start, Stop, Restart, and Check Status of Messaging Services

On Windows installations, open Task Manager > Services to start, stop, restart, and check the status of Messaging Services components.

MicroStrategy Messaging Services on Linux

Messaging Services is a component that is coupled with the Intelligence Server during installations and upgrades. Messaging Services is configured out-of-the-box and runs automatically after the installation is completed.

After installation, you can see the following services are automatically started:

  • Apache Kafka (/opt/mstr/MicroStrategy/install/MessagingServices/Kafka/kafka_2.11-0.10.1.0)
  • Apache ZooKeeper (/opt/mstr/MicroStrategy/install/MessagingServices/Kafka/kafka_2.11-0.10.1.0)
  • MicroStrategy Intelligence Server Log Consumer (/opt/mstr/MicroStrategy/install/IntelligenceServer/KafkaConsumer)

By default MicroStrategy will still send Intelligence Server diagnostic logs to local disk. Diagnostic logs will be sent to the Messaging Services Server after you perform the following:

Afterwards you will see Kafka log files created in the Kafka installation folder:

/opt/mstr/MicroStrategy/install/MessagingServices/Kafka/tmp/kafka-logs

Different Kafka topics will be created to store data for different MicroStrategy components.

Configuring Messaging Services after upgrading

By default, MicroStrategy Messaging Services are installed along with the Intelligence server upgrade.

Once you have completed the upgrade process, you need to enable MicroStrategy Messaging Services. If not, the Intelligence Server continues to write to the original log.

Messaging Services Workflow for Intelligence Server

  • Intelligence Server is the Kafka Producer and can be deployed a single node or cluster.
  • Kafka Server can be deployed as a single node or cluster.
  • Intelligence Server log consumer can run on any machine that can be connected to a Kafka Server.

Enable MicroStrategy Messaging Services

Messaging Services configuration is saved in the MicroStrategy Intelligence Server configuration. It can be enabled or disabled on the fly, without restarting your Intelligence Server.

Command Manager Scripts for Messaging Services

To check if Messaging Services is enabled, execute:

LIST ALL PROPERTIES FOR SERVER CONFIGURATION;

To enable Messaging Services through Command Manager, execute:

ALTER SERVER CONFIGURATION ENABLEMESSAGINGSERVICES TRUE CONFIGUREMESSAGINGSERVICES "bootstrap.servers:10.15.208.236:9092/batch.num.messages:5000/queue.buffering.max.ms:2000";

In the example above set:

  • bootstrap.servers: to your Kafka Server IP address and port number.
  • batch.num.messages: to the number of messages to send in one batch when using asynchronous mode.
  • queue.buffering.max.ms: to the maximum time to buffer data when using asynchronous mode.

You can specify more Kafka Producer configuration settings in this command following the same format.

Turn On the Sending Log to Messaging Services Feature

You can turn on the Sending Log to Messaging Services feature using either MicroStrategy Web or Command Manager.

From MicroStrategy Web

  1. Log in using and Administrator account.
  2. Open User Preferences > Project Defaults.
  3. Locate Sending Log to Messaging Services in the Features for Customer Feedback section.
  4. Select On from the drop-down menu.
  5. Click Apply.

From Command Manager

  1. Connect to your project source.
  2. Execute the following:

    ALTER FEATURE FLAG "SENDING LOG TO MESSAGING SERVICES" ON;

Modifying Messaging Services Configuration

Apache Kafka Server

The Kafka Server can be configured by modifying the server.properties file found in:

/opt/mstr/MicroStrategy/install/MessagingServices/Kafka/kafka_2.11-0.10.1.0

Both Apache Kafka Server and ZooKeeper should be restarted after modifying the above configuration file.

Intelligence Server Log Consumer

By default the Log Consumer is connecting to the Local Kafka Server.

There are two ways to modify the configuration of Log Consumer:

  1. Delete the LogConsumer.properties file from /opt/mstr/MicroStrategy/install/IntelligenceServer/KafkaConsumer, execute the following command, and follow the steps in the terminal:

    /opt/mstr/MicroStrategy/install/IntelligenceServer/KafkaConsumer java -jar KafkaConsumer.jar

  2. Modify file /opt/mstr/MicroStrategy/install/IntelligenceServer/KafkaConsumer/LogConsumer.properties directly.

    The default values after installation are:

    max_num_bak=1 #indicate the number of back up files

    max_file_size_M=20 #indicate the maximum file size in MB

    broker_port=9092 #Kafka Server port number

    is_silent_mode=true #indicate run consumer in slient mode

    folder_path=/opt/mstr/MicroStrategy/log/DSSErrors #indicate log folder location

    broker_hostname=127.0.0.1 #Kafka Server IP

    poll_time_out=1000 #consumer connection time out limit in seconds

MicroStrategy Messaging Services Configuration for Clustered Environments

If you have clustered your Intelligence Servers and want to use a separate machine to run MicroStrategy Messaging Services after upgrading, complete the following steps for each node in the cluster.

The minimum number of nodes for a cluster is 3.

Each node must have the following installed:

  • MicroStrategy Messaging Services
  • Apache Kafka
  • Apache Zookeeper

Configure Zookeeper

  1. Browse to folder /opt/mstr/MicroStrategy/install/MicroStrategy/MessagingServices/Kafka/kafka_2.11-0.9.0.1/config.
  2. Edit file zookeeper.properties by adding following lines:

    maxClientCnxns=0

    initLimit=5

    syncLimit=2

    server.1=10.27.20.16:2888:3888

    server.2=10.27.20.60:2888:3888

    server.3=10.15.208.236:2888:3888

    Each server parameter must contain a unique integer identifier as shown above.

  3. Go to folder /opt/mstr/MicroStrategy/install/MicroStrategy/MessagingServices/Kafka/kafka_2.11-0.9.0.1/tmp/zookeeper.

  4. Create a file named myid containing the identifying value from the server parameter name in the zookeeper.properties file.

Configure Kafka

  1. Browse to folder /opt/mstr/MicroStrategy/install/MicroStrategy/MessagingServices/Kafka/kafka_2.11-0.9.0.1/config.
  2. Edit file server.properties, add a row zookeeper.connect=10.27.20.16:2181,10.27.20.60:2181,10.15.208.236:2181 to the Zookeeper section.

    ############################# Zookeeper #############################

    # Zookeeper connection string (see zookeeper docs for details).

    # This is a comma separated host:port pairs, each corresponding to a zk

    # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

    # You can also append an optional chroot string to the urls to specify the

    # root directory for all kafka znodes.

    # zookeeper.connect=localhost:2181

    zookeeper.connect=10.27.20.16:2181,10.27.20.60:2181,10.15.208.236:2181

  3. Modify the broker.id value to a unique integer from other Kafka servers (the default value is 0), such as for node 10.27.20.60 we use number 2.

    ############################# Server Basics #############################

    # The id of the broker. This must be set to a unique integer for each broker.

    broker.id=2

Start, Stop, Restart, and Check Status of Messaging Services

Kafka Server and Zookeeper have been registered as service on Linux, so we can use service command to start, stop, and check status. The restart command is not supported.

To execute a service command for Kafka Server and Zookeeper, enter: /etc/init.d/kafka-zookeeper {start|stop|status}.

To execute a service command for MicroStrategy Intelligence Server Log Consumer, enter: /etc/init.d/consumer-iserver {stop/start/status}.