AnalysisException: Incremental stats size estimate exceeds 200.00MB

When running compute stats on large tables with many partitions you might notice an error like below.

AnalysisException: Incremental stats size estimate exceeds 200.00MB

Solution:
You should be able to add a flag within Arcengine to increase the limit size.

If you want 1GB it would be:

        --inc_stats_size_limit_bytes=1048576000

Add this option through Cloudera Manager (or Ambari) arcadia service config:
image

Note: The inc_stats_size_limit_bytes setting should be increased with care. A big value for the setting, such as 1 GB or more, can result in a spike in heap usage as well as possible crashes.

You may also wish to ensure that the java heap size can tolerate an increase in this setting as well.

1 Like