Error: Could not open log file: /var/log/arcadia/lineage/

When running Arcadia Analytics Engine (Arcengine) on a new environment you might see an error like this:

E0613 10:49:57.160387 Error flushing lineage event log: Could not open log file: /var/log/arcadia/lineage/impala_lineage_log_1.0-1560222042145
E0613 10:49:57.160544 Shutting down Arcengine Server due to abort_on_failed_lineage_event=true. Impalad exiting. 

Which usually occurs after this an error like this:

I0613 10:49:57.160656   665] TAcceptQueueServer: TServerTransport died on accept: accept(): Too many open files
I0613 10:49:57.160686   665] TServerSocket::acceptImpl() ::accept() Too many open files

If you see one or both of these errors, its likely that the Linux open file limits have not been configured properly. Resolving the issue can be done by following these instructions.