Why is Arcadia Analytics Engine (Arcengine) set to use 80% of memory and can I configure memory usage?

After installing Arcadia you will notice that there’s a memory allocation defined for the Arcadia Analytics Engine service, which may appear as a percentage for Hortonworks installs:

Its important to note that each Arcengine daemon allows you to set a process memory limit per node, and by default Arcengine will be set to use 80% of the memory on the system as it’s process memory limit unless configured differently. This does not mean that 80% of the system memory is actually available, but the Arcengine will limit itself to using only 80% of the total physical memory on the node. In cases where other processes are using some of that memory, you will tend to see more swapping from Arcengine.

For Cloudera installs this will typically be set as an explicit limit in bytes:

And in MapR or other install paths you can reference the memory limit setting (mem_limit) from the Arcengine flagfile (which you can reference from port 35000 of any Arcengine daemon):

In this example the Arcengine daemon memory limit (mem_limit) is set to 8GB:

--mem_limit=8589934592

How to configure Arcadia Analytics Engine memory limit (mem_limit):

Option 1: Configure the memory limit from the management console (available in Cloudera Manager and Ambari)

  • In Cloudera Manager, this can be set using the Memory Limit (mem_limit) in the Arcadia Configuration page:

  • In Ambari, this setting appears as a percentage, but should be configureable as an explicit number of bytes by adding mem_limit= (i.e. mem_limit=8g) to the “Optional parameters for Arcadia Analytics Engine” setting.

Option 2: Configure the memory limit using the Arcengine flagfile (MapR)

  1. Locate the arcengined flagfile for each of your nodes running the Arcadia Analytics Engine service.
  2. For each arcengined flagfile, update the --mem_limit start-up flag to be a percentage or explicit memory limit in bytes (i.e. 8gb). Memory limits can be specified as number of bytes (’<int>[bB]?’), megabytes (’<float>[mM]’), gigabytes (’<float>[gG]’), or percentage of the physical memory (’<int>%’). Defaults to bytes if no unit is given.

--mem_limit=8589934592
--mem_limit=8g
--mem_limit=8589m

or

--mem_limit=80%

1 Like