Governing Intelligence Server Memory Use with Memory Contract Manager

Memory Contract Manager (MCM) is designed to protect Intelligence Server in cases where a memory request would cause the system to approach a state of memory depletion. When enabled, MCM grants or denies requests for memory from tasks in Intelligence Server. The requests are granted or denied according to user-configured limits on the amount of memory Intelligence Server is allowed to use. Because MCM is a component in Intelligence Server, it does not manage the actual memory used by Intelligence Server itself.

MCM governs the following types of requests:

Database requests from either the MicroStrategy metadata or the data warehouse
SQL generation
Analytical Engine processing (subtotals, cross tabulation, analytic functions)
Cache creation and updating
Report parsing and serialization for network transfer
XML generation

The memory load of the requests governed by MCM depends on the amount of data that is returned from the data warehouse. Therefore, this memory load cannot be predicted.

Requests such as graphing, cache lookup, or document generation use a predictable amount of memory and, thus, are not governed by MCM. For example, a request for a report returns an acceptable amount of data. A graph of the report's results would be based on the same data and, thus, would be allowed. Therefore, MCM is not involved in graphing requests. If the report was not returned because it exceeded memory limits, the graphing request would never be issued.

Using the Memory Contract Manager

The MCM settings are in the Intelligence Server Configuration Editor, in the Governing Rules: Default: Memory Settings category.

The Enable single memory allocation governing option lets you specify how much memory can be reserved for a single Intelligence Server operation at a time. When this option is enabled, each memory request is compared to the Maximum single allocation size (MBytes) setting. If the request exceeds this limit, the request is denied. For example, if the allocation limit is set to 100 MB and a request is made for 120 MB, the request is denied, but a request for 90 MB is allowed.

If the Intelligence Server machine has additional software running on it, you may want to set aside some memory for those processes to use. To reserve this memory, you can specify the Minimum reserved memory in terms of either the number of MB or the percent of total system memory. In this case, the total available memory is calculated as the initial size of the page file plus the RAM. It is possible that a machine has more virtual memory than MCM knows about if the maximum page file size is greater than the initial size.

Intelligence Server always reserves up to 500 MB for its own operation. If the machine does not have this much memory, or if the Minimum reserved memory would leave less than 500 MB available for Intelligence Server, no memory is reserved for other processes.

When MCM receives a request that would cause Intelligence Server's memory usage to exceed the Minimum reserved memory settings, it denies the request and goes into memory request idle mode. In this mode, MCM denies any requests that would deplete memory. MCM remains in memory request idle mode until the memory used by Intelligence Server falls below a certain limit, known as the low water mark. For information on how the low water mark is calculated, see Memory Water Marks. For information about how MCM handles memory request idle mode, see Memory Request Idle Mode.

The Maximum use of virtual address space is applicable in 32-bit Windows operating systems. For 64-bit operating systems, use the Minimum reserved memory setting to control the amount of memory available for Intelligence Server.

The Memory request idle time is the longest time MCM remains in memory request idle mode. If the memory usage has not fallen below the low water mark by the end of the Memory request idle time, MCM restarts Intelligence Server. Setting the idle time to -1 causes Intelligence Server to remain idle until the memory usage falls below the low water mark.

How does MCM Grant or Deny a Request?

When a task requests memory, it provides MCM with an estimate of how much memory it requires. If the request is granted, MCM decreases the amount of available memory and the task allocates memory from the memory subsystem. When the task is completed or canceled, the memory is released and the amount of available memory increases.

MCM does not submit memory allocations to the memory subsystem (such as a memory manager) on behalf of a task. Rather, it keeps a record of how much memory is available and how much memory has been contracted out to the tasks.

A memory request is granted if it meets the following criteria:

It is smaller than the Maximum single allocation size setting.
It is smaller than the high water mark, or the low water mark if Intelligence Server is in memory request idle mode. These water marks are derived from the Intelligence Server memory usage and the Maximum use of virtual address space and Minimum reserved memory settings. For detailed explanations of the memory water marks, see Memory Water Marks.
It is smaller than 80 percent of the largest contiguous block of free memory to account for memory fragmentation.

Memory Water Marks

The high water mark (HWM) is the highest value that the sum of private bytes and outstanding memory contracts can reach before triggering memory request idle mode. The low water mark (LWM) is the value that Intelligence Server's private byte usage must drop to before MCM exits memory request idle mode. MCM recalculates the high and low water marks after every 10 MB of memory requests. The 10 MB value is a built-in benchmark and cannot be changed.

Two possible values are calculated for the high water mark: one based on virtual memory, and one based on virtual bytes. For an explanation of the different types of memory, such as virtual bytes and private bytes, see Memory.

The high water mark for virtual memory (HWM1 in the diagram above) is calculated as (Intelligence Server private bytes + available system memory). It is recalculated for each potential memory depletion.
The available system memory is calculated using the Minimum reserved memory limit if the actual memory used by other processes is less than this limit.
The high water mark for virtual bytes (HWM2 in the diagram above) is calculated as (Intelligence Server private bytes). It is calculated the first time the virtual byte usage exceeds the amount specified in the Maximum use of virtual address space or Minimum Reserved Memory settings. Because MCM ensures that Intelligence Server private byte usage cannot increase beyond the initial calculation, it is not recalculated until after Intelligence Server returns from the memory request idle state.

The high water mark used by MCM is the lower of these two values. This accounts for the scenario in which, after the virtual bytes HWM is calculated, Intelligence Server releases memory but other processes consume more available memory. This can cause a later calculation of the virtual memory HWM to be lower than the virtual bytes HWM.

The low water mark is calculated as 95 percent of the HWM. It is recalculated every time the HWM changes.

Memory Contract Management

Once the high and low water marks have been established, MCM checks to see if single memory allocation governing is enabled. If it is, and the request is for an amount of memory larger than the Maximum single allocation size setting, the request is denied.

If single memory allocation governing is not enabled, or if the request is for a block smaller than the Maximum single allocation size limit, MCM checks whether it is in memory request idle mode, and calculates the maximum contract request size accordingly:

For normal Intelligence Server operation, the maximum request size is based on the high water mark. The formula is [HWM - (1.05 *(Intelligence Server Private Bytes) + Outstanding Contracts)].
In memory request idle mode, the maximum request size is based on the low water mark. The formula is [LWM - (1.05 *(Intelligence Server Private Bytes) + Outstanding Contracts)].

The value of 1.05 is a built-in safety factor.

For normal Intelligence Server operation, if the request is larger than the maximum request size, MCM denies the request. It then enters memory request idle mode.

If MCM is already in memory request idle mode and the request is larger than the maximum request size, MCM denies the request. It then checks whether the memory request idle time has been exceeded, and if so, it restarts Intelligence Server. For a detailed explanation of memory request idle mode, see Memory Request Idle Mode.

If the request is smaller than the maximum request size, MCM performs a final check to account for potential fragmentation of virtual address space. MCM checks whether its record of the largest free block of memory has been updated in the last 100 requests, and if not, updates the record with the size of the current largest free block. It then compares the request against the largest free block. If the request is more than 80 percent of the largest free block, the request is denied. Otherwise, the request is granted.

After granting a request, if MCM has been in memory request idle mode, it returns to normal operation.

Memory Request Idle Mode

When MCM first denies a request, it enters memory request idle mode. In this mode, MCM denies all requests that would keep Intelligence Server's private byte usage above the low water mark. MCM remains in memory request idle mode until one of the following situations occurs:

Intelligence Server's memory usage drops below the low water mark. In this case, MCM exits memory request idle mode and resumes normal operation.
MCM has been in memory request idle mode for longer than the Memory request idle time. In this case, MCM restarts Intelligence Server. This frees up the memory that had been allocated to Intelligence Server tasks, and avoids memory depletion.

The Memory request idle time limit is not enforced via an internal clock or scheduler. Instead, after every denied request MCM checks how much time has passed since the memory request idle mode was triggered. If this time is more than the memory request idle time limit, Intelligence Server restarts.

This eliminates a potentially unnecessary Intelligence Server restart. For example, a memory request causes the request idle mode to be triggered, but then no more requests are submitted for some time. A scheduled check at the end of the Memory request idle time would restart Intelligence Server even though no new jobs are being submitted. However, because Intelligence Server is completing its existing contracts and releasing memory, it is possible that the next contract request submitted will be below the low water mark. In this case, MCM accepts the request and resumes normal operation, without having to restart Intelligence Server.

When MCM forces Intelligence Server to restart because of the Memory request idle time being exceeded, it also writes the contents of Intelligence Server's memory use to disk. This memory dump is saved in the file MCMServerStallDump.dmp in the Intelligence Server folder. By default, this folder is located at C:\Program Files (x86)\MicroStrategy\Intelligence Server\.

MicroStrategy recommends setting the Memory request idle time to slightly longer than the time it takes most large reports in your system to run. This way, Intelligence Server does not shut down needlessly while waiting for a task to complete. To help you determine the time limit, use Enterprise Manager to find out the average and maximum report execution times for your system. For instructions on using Enterprise Manager, see the Enterprise Manager Help.

System Memory Depletion

The diagram below shows an example of a potential depletion of system memory.

In this example, MCM grants memory request A. Once granted, a new memory contract is accounted for in the available system memory. Request B is then denied because it exceeds the high water mark, as derived from the Maximum use of virtual address space setting.

Once request B has been denied, Intelligence Server enters the memory request idle mode. In this mode of operation, it denies all requests that would push the total memory used above the low water mark.

In the example above, request C falls above the low water mark. Because Intelligence Server is in memory request idle mode, this request is denied unless Intelligence Server releases memory from elsewhere, such as other completed contracts.

Request D is below the low water mark, so it is granted. Once it has been granted, Intelligence Server switches out of request idle mode and resumes normal operation.

If Intelligence Server continues receiving requests for memory above the low water mark before the Memory request idle time is exceeded, MCM shuts down and restarts Intelligence Server.

Virtual Byte Depletion

Below is a diagram of potential memory depletion due to available bytes in the Intelligence Server virtual address space.

In this example, Intelligence Server has increased its private byte usage to the point that existing contracts are pushed above the high water mark. Request A is denied because the requested memory would further deplete Intelligence Server's virtual address space.

Once request A has been denied, Intelligence Server enters the memory request idle mode. In this mode of operation, all requests that would push the total memory used above the low water mark are denied.

The low water mark is 95 percent of the high water mark. In this scenario, the high water mark is the amount of Intelligence Server private bytes at the time when the memory depletion was first detected. Once the virtual byte high water mark has been set, it is not recalculated. Thus, for Intelligence Server to exit memory request idle mode, it must release some of the private bytes.

Although the virtual bytes high water mark is not recalculated, the virtual memory high water mark is recalculated after each request. MCM calculates the low water mark based on the lower of the virtual memory high water mark and the virtual bytes high water mark. This accounts for the scenario in which, after the virtual bytes high water mark is calculated, Intelligence Server releases memory but other processes consume more available memory. This can cause a later calculation of the virtual memory high water mark to be lower than the virtual bytes high water mark.

Intelligence Server remains in memory request idle mode until the memory usage looks like it does at the time of request B. The Intelligence Server private byte usage has dropped to the point where a request can be made that is below the low water mark. This request is granted, and MCM exits memory request idle mode.

If Intelligence Server does not free up enough memory to process request B before the Memory request idle time is exceeded, MCM restarts Intelligence Server.