When running Arcadia Analytics Engine (Arcengine) on a new environment you might see an error like this:
E0613 10:49:57.160387 Error flushing lineage event log: Could not open log file: /var/log/arcadia/lineage/impala_lineage_log_1.0-1560222042145 E0613 10:49:57.160544 Shutting down Arcengine Server due to abort_on_failed_lineage_event=true. Impalad exiting.
Which usually occurs after this an error like this:
I0613 10:49:57.160656 665 thrift-util.cc:113] TAcceptQueueServer: TServerTransport died on accept: accept(): Too many open files I0613 10:49:57.160686 665 thrift-util.cc:113] TServerSocket::acceptImpl() ::accept() Too many open files
If you see one or both of these errors, its likely that the Linux open file limits have not been configured properly. Resolving the issue can be done by following these instructions.