Why does Arcadia Analytics Engine show increases in swapping occasionally?

Memory swapping occurs when large amounts of memory are being consumed by the host. The decision of whether to use swap or not is done by the kernel, rather than the Arcadia Analytics Engine. Typically Arcadia Analytics Engines running on a node that is reaching its configured allocated mem_limit setting, queries will error out with a “memory limit exceeded error.”

From Cloudera:
https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_performance.html#cdh_performance__section_xpq_sdf_jq

The Linux kernel parameter, vm.swappiness, is a value from 0-100 that controls the swapping of application data (as anonymous pages) from physical memory to virtual memory on disk. The higher the value, the more aggressively inactive processes are swapped out from physical memory. The lower the value, the less they are swapped, forcing filesystem buffers to be emptied.

On most systems, vm.swappiness is set to 60 by default. This is not suitable for Hadoop clusters because processes are sometimes swapped even when enough memory is available. This can cause lengthy garbage collection pauses for important system daemons, affecting stability and performance.

It is recommended that you set vm.swappiness to a value between 1 and 10, preferably 1, for minimum swapping.

How can we decrease swapping?

Adding more memory to your hosts to accomodate your workloads at peak processing hours, or adding additional hosts to spread workloads and memory usage.

1 Like