Governing Intelligent Cube Memory Usage

Intelligent Cubes must be stored in Intelligence Server memory for reports to access the data. While this can improve performance of these reports, loading too much data onto Intelligence Server memory can negatively affect Intelligence Server's ability to process jobs. For this reason, it is important to limit how much Intelligent Cube data can be stored on Intelligence Server.

Determining Memory Limits for Intelligent Cubes

Storing Intelligent Cubes can greatly improve performance by allowing reports to execute against an in memory copy of data. However, storing too much Intelligent Cube data in memory can cause other Intelligence Server processes to compete for system resources and may cause degradations in performance. This makes defining a memory limit for Intelligent Cubes an important step in maintaining Intelligence Server response time.

An Intelligent Cube memory limit defines the maximum amount of RAM of the Intelligence Server machine that can be used to store loaded Intelligent Cubes. This data is allocated separately of memory used for other Intelligence Server processes.

For example, you define a memory limit on Intelligent Cubes to be 512 MB. You have 300 MB of Intelligent Cube data loaded into Intelligence Server memory, and normal processing of other Intelligence Server tasks uses 100 MB of memory. In this scenario, Intelligence Server uses 400 MB of the RAM available on the Intelligence Server machine. This scenario demonstrates that to determine a memory limit for Intelligent Cubes, you must consider the below factors:

• The amount of RAM available on the Intelligence Server machine, and of that RAM what percentage can be used by Intelligence Server without negatively affecting performance and successful operation of the host machine.
• The average and peak usage of RAM by Intelligence Server processes other than Intelligent Cube storage. For information on setting governing limits on other Intelligence Server processes and monitoring system usage, Governing Intelligent Cube Memory Usage, Loading, and Storage.
• The amount of memory required to load all Intelligent Cubes necessary to meet reporting requirements. To help save space, Intelligent Cubes that are not used often can be unloaded until they are required by reports (see Monitoring and Modifying Intelligent Cube Status).
• The Maximum RAM usage (Mbytes) memory limit can be defined per project. If you have multiple projects that are hosted from the same Intelligence Server, each project may store Intelligent Cube data up to its memory limit.
• For example, you have three projects and you set their Maximum RAM usage (Mbytes) limits to 1 GB, 1 GB, and 2 GB. This means that 4 GB of Intelligent Cube data could be stored in RAM on the Intelligence Server machine if all projects reach their memory limits.
• The size of the Intelligent Cubes that are being published and loaded into memory. The process of publishing an Intelligent Cube can require memory resources in the area of two to four times greater than the Intelligent Cube's size. This can affect performance of your Intelligence Server and the ability to publish the Intelligent Cube. For information on how to plan for these memory requirements, see the next section.
• If your project and Intelligent Cubes support multiple languages, each language supported may require additional memory.
• To help reduce Intelligent Cube memory size, review the best practices described in Best Practices for Reducing Intelligent Cube Memory Size.

Best Practices for Reducing Intelligent Cube Memory Size

MicroStrategy recommends the following best practices to reduce the memory size of your Intelligent Cubes:

• Attributes commonly use numeric values for their ID forms. Using attributes defined in this way can save space as compared to attributes that use character strings for their ID forms.
• Attribute forms should be included only as required because including additional attribute forms in Intelligent Cubes requires additional memory.
• You should avoid including compound metrics and smart metrics in Intelligent Cubes. The same results provided by compound metrics and smart metrics can often be provided by creating derived metrics in reports that connect to Intelligent Cubes.
• You can define Intelligent Cube normalization to reduce the amount of memory required for an Intelligent Cube. Intelligent Cube normalization can be configured using the Data population for Intelligent Cubes VLDB property. For instructions on setting VLDB properties, see SQL Generation and Data Processing: VLDB Properties.

Planning for Intelligent Cube Publishing and Loading

Publishing an Intelligent Cube can require memory resources in the area of two to four times greater than the size of an Intelligent Cube. Once the Intelligent Cube is published, the extra resources are returned to the system and only the space required for the Intelligent Cube and some indexing information is taken up in RAM. However, you should consider this peak in memory usage when planning for the publication of Intelligent Cubes.

By default, publishing an Intelligent Cube includes the step of loading the Intelligent Cube into memory. You can modify this default behavior as described in Monitoring and Modifying Intelligent Cube Status. Loading and unloading Intelligent Cubes is described in Monitoring and Modifying Intelligent Cube Status.

If publishing an Intelligent Cube is processed in RAM alone without using swap space, this can greatly reduce the effect publishing an Intelligent Cube has on the performance of your Intelligence Server host machine. Swap space is controlled by the operating system of a computer and using this space for the transfer of data into RAM can negatively affect the performance of a computer.

You can help to keep the processes of publishing Intelligent Cubes within RAM alone by defining memory limits for Intelligent Cubes that reflect your Intelligence Server host's available RAM as well as schedule the publishing of Intelligent Cubes at a time when RAM usage is low. For information on scheduling Intelligent Cube publishing, see the In-memory Analytics Help.

To determine memory limits for Intelligent Cubes, you should review the considerations listed in Determining Memory Limits for Intelligent Cubes. You must also account for the potential peak in memory usage when publishing an Intelligent Cube, which can be two to four times the size of an Intelligent Cube.

For example, your Intelligence Server machine has 2 GB of RAM and 2 GB of swap space. Assume that normal usage of RAM by the operating system uses .4 GB of RAM. This leaves a possible 1.6 GB of RAM for Intelligent Cube storage and other tasks.

With this configuration, consider the act of publishing a 1 GB Intelligent Cube. Assuming the peak in memory usage for publishing this Intelligent Cube is 2.5 times the size of the Intelligent Cube, the publishing process requires 2.5 GB. This can take up 1.6 GB of RAM, but it also requires .9 GB of swap space, as shown in the diagram below.

Once the Intelligent Cube is published, only the 1 GB for the Intelligent Cube (plus some space for indexing information) is used in RAM and the remaining .6 GB of RAM and .9 GB of swap space used during the publishing of the Intelligent Cube is returned to the system, as shown in the image below.

While the Intelligent Cube can be published successfully, using the swap space could have an affect on performance of the Intelligence Server machine.

With the same configuration, consider the act of publishing a .5 GB Intelligent Cube rather than a 1 GB Intelligent Cube. Assuming the peak in memory usage for publishing this Intelligent Cube is 2.5 times the size of the Intelligent Cube, the publishing process requires 1.25 GB. This process can be handled completely within RAM, as shown in the diagram below.

Once the Intelligent Cube is published, only the .5 GB for the Intelligent Cube (plus some space for indexing information) is used in RAM and the remaining RAM used during the publishing of the Intelligent Cube is returned to the system, as shown in the image below.

Be aware that as more Intelligent Cube data is stored in RAM, less RAM is available to process publishing an Intelligent Cube. This along with the peak memory usage of publishing an Intelligent Cube and the hardware resources of your Intelligence Server host machine should all be considered when defining memory limits for Intelligent Cube storage per project.

Defining Memory Limits for Intelligent Cubes

You can define limits for the amount of Intelligent Cube memory stored in Intelligence Server at a given time in two ways described below:

• You can use the amount of data required for all Intelligent Cubes to limit the amount of Intelligent Cube data stored in Intelligence Server memory at one time for a project. The default is 256 megabytes.

  The total amount of memory used on Intelligence Server by Intelligent Cubes for a project is calculated and compared to the limit you have defined. If an attempt to load an Intelligent Cube is made that will exceed this limit, an Intelligent Cube is removed from Intelligence Server memory before the new Intelligent Cube is loaded into memory.

• You can use the number of Intelligent Cubes to limit the number of Intelligent Cube stored in Intelligence Server memory at one time for a project. The default is 1000 Intelligent Cubes.

  The total number of Intelligent Cubes for a project that are stored in Intelligence Server memory is compared to the limit you define. If an attempt to load an Intelligent Cube is made that will exceed the numerical limit, an Intelligent Cube is removed from Intelligence Server memory before the new Intelligent Cube is loaded into memory.

To Define Limits on Intelligence Server Memory Usage by Intelligent Cubes

1. In Developer, log in to a project that uses Intelligent Cubes. You must log in using an account with the Administer Cubes privilege.
2. Right-click the project and select Project Configuration.
3. Expand Intelligent Cubes, and then select General.
4. Define the values for the following project governing options:
   • Maximum RAM usage (MBytes): Defines the data size, in megabytes, to limit the amount of Intelligent Cube data that can be stored in Intelligence Server memory for a project at one time. The default value is 256 megabytes.

   • Maximum number of cubes: Defines the maximum total number of Intelligent Cubes that can be created for a project, including Intelligent Cubes that are not loaded into Intelligence Server's memory. The default value is 1000 Intelligent Cubes.

     Beginning with MicroStrategy 2020 Update 1, this governing setting is being retired. It will remain available, but the setting will not be enforced if set below the default value of 1000.

   • Maximum cube size allowed for download (MB): Defines the maximum size of a cube, in megabytes. The default is 200.
5. Click OK.

Defining Limits for Intelligent Cube Indexes

Intelligence Server generates indexes to speed up access to data in Intelligent Cubes. In very large Intelligent Cubes, these indexes may significantly increase the size of the Intelligent Cube. You can define limits for how much the indexes can add to the size of the Intelligent Cube at the project level, using the Project Configuration Editor.

To Define Limits for the Intelligent Cube Indexes

1. In Developer, log in to the project source for your project.
2. In the Folder List, right-click the project and choose Project Configuration.
3. Expand Intelligent Cubes, and then select General.
4. Edit the following values, as appropriate:
   • Maximum % growth of an Intelligent Cube due to indexes: Defines the maximum that indexes are allowed to add to the Intelligent Cube's size, as a percentage of the original size. For example, a setting of 50 percent defines that a 100 MB Intelligent Cube can grow to 150 MB because its indexes. If the Intelligent Cube's size exceeds this limit, the least-used indexes are saved to a disk and then deleted from memory sequentially until the cube size is below the upper limit.
   • Cube growth check frequency (in mins): Defines, in minutes, how often the Intelligent Cube's size is checked, and if necessary, how often the least-used indexes are dropped.
5. Click OK.

Defining Limits for Intelligent Cubes Created using the Import Data Feature

The Import Data feature allows users to create Intelligent Cubes using external data sources, such as Excel files, tables in a database, or Freeform SQL queries. You can define limits on the size of files that users can upload, and on the total memory available for the Intelligent Cubes they can create.

To Define Limits on Intelligent Cubes Created Using the Import Data Feature

1. In Developer, log in to a project using an account with the Administer Cubes privilege.
2. In the Folder List, right-click on the project and select Project Configuration.
3. Expand Governing Rules, expand Default, and then select Import Data.
4. Define values for the following options:
   • Maximum file size (MB): Defines the maximum size for files that users can upload and import data from. The default value is 30 MB. The minimum value is 1 MB, and the maximum value is 9999999 MB.
   • Maximum quota per user (MB): Defines the maximum size of all data import cubes for each individual user. This quota includes the file size of all data import cubes, regardless of whether they are published to memory or on disk. You can set the maximum size quota by entering the following values:
     • -1: Unlimited - No limit is placed on the size of data import cubes for each user.
     • 0: Default - The default size limit of 100 MB is applied to each user
     • [1, ∞): Specific limit - Entering a value of 1 or greater will apply a MB limit of that size to each user.

       In a clustered environment, this setting applies to all nodes in the cluster.

5. Click OK.