Arcengine crashes upon startup repeatedly

  1. Check the arcengine.INFO & ERROR logs in /var/log/arcadia

  2. Check if the dump is generated usually it should appear in the logs.

    ...
    --statestore_num
    Wrote minidump to /var/log/arcadia/minidumps/arcengined/790e7b89-5db1-4016-16a6188f-0c848b07.dmp
    ....
    
  3. Is the same issues observed with other components: statestore, catalog, arcviz
    Usually what we see is that catalog & arcengine crash/generate mini dumps while statestore and arcviz will come fine

  4. Use string to extract information about libraries being loaded:

    strings arcengined.dump

    The output will look like the below.

    If you notice extraneous libraries being loaded such as /opt/Panorama/hedzup/mn/lib/librpilj64.so then it’s very likely some security software that might be intercepting the loading of the process and forcing certain libraries to be dynamically load, resulting in the crash.

     00400000-0280e000 r-xp 00000000 ca:02 377494248                          /opt/cloudera/parcels/ARCADIAENTERPRISE-4.2.1.1-1525418832.cdh5/lib/arcengine/bin/arcengined
     0280e000-0295d000 rw-p 0240d000 ca:02 377494248                          /opt/cloudera/parcels/ARCADIAENTERPRISE-4.2.1.1-1525418832.cdh5/lib/arcengine/bin/arcengined
     0295d000-029af000 rw-p 00000000 00:00 0 
     03a00000-04400000 rw-p 00000000 00:00 0                                  [heap]
     7fadeca0f000-7fadf2f38000 r--p 00000000 ca:02 8657028                    /usr/lib/locale/locale-archive
     7fadf2f38000-7fadf2f74000 r-xp 00000000 ca:02 360784541                  /opt/Panorama/hedzup/mn/lib/librpilj64.so
     7fadf2f74000-7fadf3174000 ---p 0003c000 ca:02 360784541                  /opt/Panorama/hedzup/mn/lib/librpilj64.so
     7fadf3174000-7fadf317e000 rw-p 0003c000 ca:02 360784541                  /opt/Panorama/hedzup/mn/lib/librpilj64.so
     7fadf317e000-7fadf3180000 rw-p 00000000 00:00 0 
     7fadf3180000-7fadf3181000 ---p 00000000 00:00 0 
     7fadf3181000-7fadf3981000 rw-p 00000000 00:00 0 
     7fadf3981000-7fadf3982000 ---p 00000000 00:00 0 
     7fadf3982000-7fadf4182000 rw-p 00000000 00:00 0 
     7fadf4182000-7fadf4183000 ---p 00000000 00:00 0 
     7fadf4183000-7fadf4983000 rw-p 00000000 00:00 0 
     7fadf4983000-7fadf4984000 ---p 00000000 00:00 0 
     7fadf4984000-7fadf5184000 rw-p 00000000 00:00 0 
     7fadf5184000-7fadf5188000 r-xp 00000000 ca:02 8781625                    /usr/lib64/sasl2/libplain.so.3.0.0
     7fadf5188000-7fadf5387000 ---p 00004000 ca:02 8781625                    /usr/lib64/sasl2/libplain.so.3.0.0
     7fadf5387000-7fadf5388000 r--p 00003000 ca:02 8781625                    /usr/lib64/sasl2/libplain.so.3.0.0
     7fadf5388