MicroStrategy ONE

Configure Platform Analytics Using the PAConsumerConfig.yaml File

Platform Analytics stores all configuration parameters for the Telemetry Store (previously Platform Analytics Consumer) and the Identity Telemetry producer (previously Usher Metadata Producer) in the PAConsumerConfig.yaml file. For more information about Platform Analytics architecture reference, see Platform Analytics Architecture and Services.

The YAML file structure is updated each release with new configuration parameters or Telemetry Server topics. All modifiable values are retained after an upgrade, so any customized parameters are not lost. However, all newly‑added fields are set to the default after an upgrade.

The YAML file is located on the machine where Platform Analytics was installed using the MicroStrategy Installation Wizard.

The default path is:

  • Linux: /opt/MicroStrategy/PlatformAnalytics/Conf
  • Windows: C:\Program Files (x86)\MicroStrategy\Platform Analytics\conf

How to Read a YAML File

In a YAML file, indentation is used to express nested values. For example:

Copy 
parentConfig:
  numberOfConsumers: 1
  pollTimeoutMillisec: 1000
  kafkaProperties:
    bootstrap.servers: "10.27.17.167:9092"

YAML uses the key: value notation. A single space is required after the colon.

To read more about YAML functionality, see Learn YAML in Y minutes.

PAConsumerConfig.yaml Specifications

The PAConsumerConfig file consists of the following parts:

  • paParentConfig: Common configurations for the Telemetry Server (Kafka) and Telemetry Manager (Zookeeper) across TopicsGroups
  • paEtlConfig: Configuration for the Telemetry Store (Platform Analytics Consumer) to perform data processing.
  • usherServerConfig: Connectivity configuration parameters for connecting to the Identity Server database to collect Identity metadata information.
  • paTopicsGroupList: List of Telemetry Server TopicsGroups and their configuration.

Each topicsGroup inherits settings from defaultConfig and parentConfig. Each topicsGroup can also override specific settings that it inherits.

Below is a sample PAConsumerConfig.yaml file. Reference the installed file for the latest version.

Copy 
---
paParentConfig:
  consumerGroupSuffix: ~
  overrideKafkaOffsets: true
  kafkaTopicNumberOfReplicas: 1
  kafkaTopicsDoNotCreateList:
  zooKeeperConnection: 127.0.0.1:2181
  ignoreUsherTopics: false
  kafkaConsumerProperties:
    bootstrap.servers: 127.0.0.1:9092

paEtlConfig:
  redisConnection: 
    redisServer: 127.0.0.1
    redisPort: 6379
    redisPassword:
  dailyETLConfiguration:
    scheduleHour: 5
    scheduleMin: 2
  viewCutoffRangeInDays: 14
  beaconDedup: true
  locationDedup: true
  warehouseDbConnection:
    whHost: 127.0.0.1
    whUser: root
    whPasswd:
    whPort: 3306
    whDb: platform_analytics_wh
    whClientCertificateKeyStore:
    whClientCertificateKeyStoreType:
    whClientCertificateKeyStorePassword:
    whTrustCertificateKeyStore:
    whTrustCertificateKeyStoreType:
    whTrustCertificateKeyStorePassword:
  pgWarehouseDbConnection:
    pgWhHost: localhost
    pgWhUser: mstr_pa
    pgWhPasswd: Ugjx+93ROzBsA2gwBOWT5Qlu6hbfg5frTBmLmg==,970sBwUbi4EowB/4
    pgWhPort: 5432
    pgWhDb: platform_analytics_wh
    pgWhSSLcert: ~
    pgWhSSLkey: ~
    pgWhSSLrootcert: ~
    pgWhSSLmode: ~
  geoLocationTopic: Mstr.PlatformAnalytics.Geolocation
  kafkaHealthCheckTopic: mstr-pa-health-check
  usherProducerKeys:
    - SourceProvisionBadgePhone
    - SourceProvisionOrganization
    - SourceEnvironmentVariables
    - SourceOrganization
    - SourceOrganizationBadge
    - SourceBadgeAdminRole
    - SourceBadge
    - SourceGateway
    - SourceGatewayHierarchyAndDef
    - SourceBeacon
    - SourceDevice
  googleAPIConfig:
    googleApiKey:
    googleApiClientId:
    businessQuota: 100000
    freeQuota: 2500
    sleepTimeQuery: 5
  usherLookupTopic: Mstr.PlatformAnalytics.UsherLookup

usherServerConfig:
  usherServerDbConnection:
    usherServerMysqlAesKeyPath:
    usherServerUrl:
    usherServerUser:
    usherServerPassword:

paTopicsGroupList:
  -
    name: UsherInboxMessage
    numberOfConsumers: 1
    usherFlag: true
    topics:
      - Mstr.IdentityServer.ActionLog
  -
    name: UsherInboxResponse
    numberOfConsumers: 1
    usherFlag: true
    topics:
      - Mstr.IdentityServer.ActionLog
  -
    name: Geolocation
    numberOfConsumers: 1
    usherFlag: true
    topics:
      - Mstr.PlatformAnalytics.Geolocation
  -
    name: UsherLog
    numberOfConsumers: 2
    usherFlag: true
    topics:
      - Mstr.IdentityServer.ActionLog
      - Mstr.IdentityServer.LocationLog

paParentConfig Settings

The paParentConfig settings are common configurations for the Telemetry Server (Kafka) and Telemetry Manager (Zookeeper) across TopicsGroups. For example:

Copy 
---
paParentConfig:
  consumerGroupSuffix: ~
  overrideKafkaOffsets: true
  kafkaTopicNumberOfReplicas: 1
  kafkaTopicsDoNotCreateList:
  zooKeeperConnection: 127.0.0.1:2181
  ignoreUsherTopics: false
  kafkaConsumerProperties:
    bootstrap.servers: 127.0.0.1:9092

Below are settings defined for both the paParentConfig and topicsGroup configuration, along with the defaultConfig values for each setting.

Name Default Description
consumerGroupSuffix ~ (null value)

This field is used for testing or recovering data in a production environment. Appended to the topicsGroup name makes up the actual Consumer Group ID (also referred to as group.id).

A key use for this field is to change the property in order to generate a new Consumer Group ID. Generating a new Consumer Group ID causes all records remaining in the Telemetry Server topics to be reprocessed.

For example, in order to repopulate the Platform Analytics Warehouse (PA WH), you can modify this property to a unique string after re-initializing the PA WH by using the PA custom installer. The unique string should be a string not used before in the same environment. A recommended strategy is to include a timestamp in it.

For example 

reprocess_incorrect_log_johndoe_1330111282018
overrideKafkaOffsets true

If true, uses topic-partition offset values in the database to set the Kafka offsets for a given Consumer Group at startup.

If false, uses offset values stored in Kafka.

It's recommended to keep the configuration as default.
kafkaTopicNumberOfReplicas 1 or number of Telemetry Servers

This is the replica factor configured for all Telemetry Server topics. It is set during the installation of Platform Analytics depending on if a cluster of Telemetry Servers is installed or a single node.

This value should match the number of Telemetry Server nodes clustered in order to take advantage of Kafka's failure tolerance.

kafkaTopicsDoNotCreateList empty string The list of topics under topicsGroupList that will not be created by the Telemetry Store (Platform Analytics Consumer) upon startup. This field should not be modified.
zooKeeperConnection 127.0.0.1:2181 or pre‑configured Zookeeper cluster quorum

The comma-separated Telemetry Manager (Zookeeper) cluster configuration.

For example: FQDN1:PORT1, FQDN2:PORT2, FQDN3:PORT3

The default port is 2181, set during the installation of Platform Analytics.

ignoreUsherTopics false

This value is set during the installation of Platform Analytics depending on if the Identity Server was installed or not.

false if Identity Server is installed and configured;

true if Identity Server is not installed.

bootstrap.servers 127.0.0.1:9092 or pre‑configured Kafka broker quorum

The comma-separated Telemetry Server (Kafka) cluster configuration (e.g. FQDN1:PORT1, FQDN2:PORT2, FQDN3:PORT3).

The default port is 9092, set during the installation of Platform Analytics.

paEtlConfig Settings

Copy 
paEtlConfig:
    redisConnection:
        redisServer: 127.0.0.1
        redisPort: 6379
        redisPassword: ~
    dailyETLConfiguration:
        scheduleHour: 5
        scheduleMin: 2
    viewCutoffRangeInDays: 14
    currentFactDataKeepDays: 180
    beaconDedup: true
    locationDedup: true 
    whDbType: postgresql
    warehouseDbConnection:
        whHost: 127.0.0.1
        whUser: root
        whPasswd: r9oJP5d6
        whPort: 3306
        whDb: platform_analytics_wh
    pgWarehouseDbConnection:
        pgWhHost: localhost
        pgWhUser: mstr_pa
        pgWhPasswd: Ugjx+93ROzBsA2gwBOWT5Qlu6hbfg5frTBmLmg==,970sBwUbi4EowB/4
        pgWhPort: 5432
        pgWhDb: platform_analytics_wh
        pgWhSSLcert: ~
        pgWhSSLkey: ~
        pgWhSSLrootcert: ~
        pgWhSSLmode: ~
    geoLocationTopic: Mstr.PlatformAnalytics.Geolocation
    kafkaHealthCheckTopic: mstr-pa-health-check
    usherProducerKeys:
        - SourceProvisionBadgePhone
        - SourceProvisionOrganization
        - SourceEnvironmentVariables
        - SourceOrganization
        - SourceOrganizationBadge
        - SourceBadgeAdminRole
        - SourceBadge
        - SourceGateway
        - SourceGatewayHierarchyAndDef
        - SourceBeacon
        - SourceDevice
    googleAPIConfig:
        googleApiKey: ~
        googleApiClientId: ~ 
        businessQuota: 100000
        freeQuota: 2500
        sleepTimeQuery: 5
    usherLookupTopic: Mstr.PlatformAnalytics.UsherLookup

Below are settings defined for paEtlConfig, along with the defaultConfig values for each setting.

Name Default Description
redisServer 127.0.0.1

The fully qualified domain name (FQDN) or IP for the Telemetry Cache (Redis server).

For best performance, use a local Telemetry Cache instance.

redisPort 6379

The port for Telemetry Cache (Redis server).

The default is 6379, set during installation.

redisPassword empty string

The password to connect to the Telemetry Cache (Redis server) if password authentication is enabled.

By default, password authentication is not enabled.

scheduleHour 5

The hour specified for the Platform Analytics daily ETL to kick off. The default value 5 means 05:00 UTC.
scheduleMin 2 The minute of the scheduled hour for the Platform Analytics daily ETL to run. The default value 2 stands for 2 minutes past the scheduled hour.
viewCutoffRangeInDays 14

The number of days of data for which the view tables in the Platform Analytics Cube will hold in memory during republish.

For example, a 14- day default means that the view tables and Platform Analytics Cube will include data from the last rolling 14 days. The data returned by the Platform Analytics project schema is never limited.

For more details, see Modify the Amount of Data Returned In-Memory for the Platform Analytics Cube.
currentFactDataKeepDays 180

The number of days of data which the current fact tables in the Platform Analytics Repository will hold.

For the PostgreSQL warehouse, we create the historical table for some fact tables whose data amount may be very large. Like access_transactions, fact_sql_stats. The historical table' name starts with the prefix "historical_".

For example, a 180- day default means that the current fact tables will include data from the last rolling 180 days. And all other data will be stored in the historical fact tables.

beaconDedup true

A flag for determining whether de-duplication of the MicroStrategy Badge beacon tracking data is turned on.

If true, the Telemetry Store ETL will remove any duplicate beacon actions if all of the conditions are met:

log from the same user

interacting with the same beacon

within 180 seconds

Turning this flag on helps keep the minimum valid data points for analysis without excess data collection.

locationDedup true

A flag for determining whether de-duplication of the MicroStrategy Badge location tracking data is turned on.

If true, the Telemetry Store ETL removes any duplicate location tracking actions if all of the conditions are met:

log from the same user

within 60 seconds

Turning this flag on will help keep the minimum valid data points for analysis without excess data collection.

whDbType

postgresql

The default database type that was used as the Platform Analytics Repository. Beginning with MicroStrategy 2020, the default database is "postgresql" but it can also support database type "mysql".
whHost pre‑configured via installation The fully qualified domain name (FQDN) or IP of the Platform Analytics Repository where the Telemetry Store will store data for reporting.
whUser pre‑configured via installation The username used to connect to the Platform Analytics Repository where the Telemetry Store will store data for reporting.
whPasswd pre‑configured via installation The password of the user used to connect to the Platform Analytics Repository where the Telemetry Store will store data for reporting.
whPort 3306

The port of the MySQL database server to connect .

The port for Platform Analytics Repository (MySQL Server Database).

The default is 3306, set during installation.

whDb platform_analytics_wh

The database of the Platform Analytics warehouse.

This should not be changed.

pgWhHost

localhost

The fully qualified domain name (FQDN) or IP of the PostgreSQL database which used for Platform Analytics Repository. As we will install the PostgreSQL Server on the machine which installed the Platform Analytics, the default value is "localhost"

pgWhUser

mstr_pa

The PostgreSQL database username used to connect to the Platform Analytics Repository where the Telemetry Store will store data for reporting.

pgWhPasswd

pre‑configured via installation

The PostgreSQL database password of the user used to connect to the Platform Analytics Repository where the Telemetry Store will store data for reporting. This password is encrypted during the installation. You can find the unencrypted password from file "Default_Accounts.txt" which under the (Windows: C:\Program Files (x86)\Common Files\MicroStrategy\ or Linux: ./install/Repository/)

pgWhPort

5432

The port of the PostgreSQL database server to connect.

The port for Platform Analytics Repository (PostgreSQL Server Database).

The default is 5432, set during installation.

pgWhDb

platform_analytics_wh

The database of the Platform Analytics warehouse.

This should not be changed.
pgWhSSLcert empty string For future SSL authentication support.
pgWhSSLkey empty string For future SSL authentication support.
pgWhSSLrootcert empty string For future SSL authentication support.
pgWhSSLmode empty string For future SSL authentication support.
geoLocationTopic Mstr.PlatformAnalytics.Geolocation

The Telemetry Server (Kafka) topic for location data geocoding processing from the MicroStrategy Badge mobile app.

This should not be changed.

kafkaHealthCheckTopic mstr-pa-health-check

The Telemetry Server (Kafka) topic used for the health check.

This should not be changed.

usherProducerKeys

SourceProvisionBadgePhone

SourceProvisionOrganization

SourceEnvironmentVariables

SourceOrganization

SourceOrganizationBadge

SourceBadgeAdminRole

SourceBadge

SourceGateway

SourceGatewayHierarchyAndDef

SourceBeacon

SourceDevice

 This should not be changed.
logging True Flag for determining if the Google geocoding API usage logging is enabled.
alerting True Flag for determining if the Google geocoding API usage logging is enabled.
googleApiKey empty string

The business key to allow making Google geocoding API calls with a business quota.
googleApiClientId empty string The business key to allow making Google geocoding API calls with a business quota.
businessQuota 100000 The daily quota for making Google geocoding API calls without any developer or business keys.
callLimit 1000 For internal use only.
sleepTimeQuery 5

The number of seconds to pause between the Google geocoding API calls for location data processing.

This should not be changed.

usherLookupTopic Mstr.PlatformAnalytics.UsherLookup

The Kafka topic used for Usher server metadata information telemetry.

This should not be changed.

usherServerConfig Settings

Copy 
usherServerConfig:
  usherServerDbConnection:
    usherServerMysqlAesKeyPath:
    usherServerUrl:
    usherServerUser:
    usherServerPassword:

Below are settings defined for usherServerConfig, along with the defaultConfig values for each setting.

Name Default Description
usherServerMysqlAesKeyPath pre‑configured via installation The AES key file path used to decrypt the password
usherServerUrl pre‑configured via installation The JDBC connectivity URL to connect to Usher Server meta information database
usherServerUser pre‑configured via installation The username to connect to Usher Server meta information database
usherServerPassword pre‑configured via installation The password to connect to Usher Server meta information database

paTopicsGroupList Settings

The following settings that are defined only at the topicsGroup level, not at the ParentConfig.

Copy 
paTopicsGroupList:
  -
    name: UsherInboxMessage
    numberOfConsumers: 1
    usherFlag: true
    topics:
      - Mstr.IdentityServer.ActionLog
  -
    name: UsherInboxResponse
    numberOfConsumers: 1
    usherFlag: true
    topics:
      - Mstr.IdentityServer.ActionLog
  -
    name: Geolocation
    numberOfConsumers: 1
    usherFlag: true
    topics:
      - Mstr.PlatformAnalytics.Geolocation
  -
    name: UsherLog
    numberOfConsumers: 2
    usherFlag: true
    topics:
      - Mstr.IdentityServer.ActionLog
      - Mstr.IdentityServer.LocationLog

Below are settings defined for paTopicsGroupList.

Name Description
name Name for the topicsGroup. Must be unique among all topicsGroup names.
numberOfConsumers The number of consumer processes assigned to this topicsGroup when forming a consumer group
usherFlag true if this topicGroup is Usher-related.
topics The list of Kafka topics that the consumers in the topicsGroup subscribe to.