Arcengine Session limits - how to avoid hanging queries and freeing up sessions & resources

Session limits at the Arcadia Engine level are critical to the operation of the system. If session limits are hit, it can pause new queries from running, or put the arcviz process in a hanging state and most pages might not respond at all.

A few configuration changes can help with this:

1) Increase session limit

Current session limits for each arcengine coordinator node is 64. Bump this up using the following two configs:

 --be_service_threads=512
 --fe_service_threads=512

(this would applied in Cloudera Manger / Ambari through the Arcadia Analytics Engine flagfile)

2) Close idle session

To help close idle sessions established by 3rd party tools like Tableau/MSTR and avoid taking up resources when session or queries become idle, the recommended Arcengine setting in seconds is:

--idle_session_timeout=300
--idle_query_timeout=300**

(this would applied in Cloudera Manger / Ambari through the Arcadia Analytics Engine flagfile similar to above)

You can then confirm these limit increased after restarting arcadia service, go to port 35000 varz page, and searching for these settings.