Failed to create thread... Resource temporarily unavailable

If your Linux nproc security limits setting is too low for Arcadia you may see an error like this once more users enter the platform:

Visual XXXX - Could not load data
error: (HY000, None) Failed to create thread SenderThread(1:1) 
in category DataStreamSender: boost::thread_resource_error: 
Resource temporarily unavailable

To fix this issue you will need to increase the nproc limits for the “arcadia” user on each node running Arcadia services:

arcadia          soft    nproc     65536
arcadia          hard    nproc     65536

There’s another scenario similar to the above where LLVM in Arcengine hits a fatal error when it hits the nproc thread limit.

F0629 08:20:02.956413 29088 [llvm-codegen.cc:111](http://llvm-codegen.cc:111/)] LLVM hit fatal error: Unable to allocate section memory!

Its suggested that if this error occurs that you increase nproc once again (In /etc/security/limits.conf or a specific conf file for arcadia depending on your setup):

arcadia soft nproc 262144
arcadia hard nproc 262144

And also increase these linux proc configurations on all the nodes running Arcadia Analytics Engine (Arcengine):

echo 2000000 > /proc/sys/kernel/threads-max
echo 2000000 > /proc/sys/kernel/pid_max
echo 8000000 > /proc/sys/vm/max_map_count

To persist the above changes you can edit the /etc/sysctl.conf file and add/modify these settings:

kernel.threads-max=2000000
kernel.pid_max=2000000
vm.max_map_count=8000000

Run the following command when you completed and saved the changes.

sysctl -p