Unsupported type 'DATE' errors while creating a Dataset

When creating Arcadia Datasets for the first time, you may encounter errors like this:

15%20AM

The DATE type is strictly a Hive data type, which means its only supported when the data is queried directly from a Hive connection. You can modify your table schema’s slightly to use STRING or TIMESTAMP data types to get around this issue, which are valid data types in Arcadia and more widely supported with other query engines (i.e. Impala).

We use DATE type in Hive in almost all of our database tables. We do not intend to amend this data types in Hive. Could you please arrange to support the DATE type in Arcadia.

@trit there are currently no plans to support Hive DATE type. STRING & TIMESTAMP are more operable and widely used hence why we support those.

Ca you at least provide some workaround (like auto converting to string) to allow people to read the data in the DATE column? I believe, most people would like to keep the existing schema. Thanks for considering.

1 Like

@Dao-I_Tony_Lin, you can create an external table definition based on the schema in your original table and then substitute STRING types for the DATE types. You would also point the external table to the location of the files from your first table (SHOW CREATE TABLE should give you location information), and you will also need to make sure that you run MSCK REPAIR <db>.<new external tablename> on your external table when you add partitions or make updates to your original Hive table.

1 Like

@trit one follow up question - are you tables stored in ORC? if so, we should be able to support them in our latest release 4.5.

Can you confirm your storage format and version of Arcadia?

1 Like

@Dao-I_Tony_Lin Can you confirm your storage format and version of Arcadia?

Our tables are stored either in Parquet or ORC. Currently, we use Arcadia Enterprise 4.4.0.1

@trit if you upgrade to Arcadia Enterprise 4.5 and you have Date types in ORC, it should work.

Example ORC table with “DATE” column type :

      CREATE TABLE shaun.date_orc (a int, b DATE)
      partitioned by (yyyymm INT) stored as ORC; 

Inside Arcadia 4.5 you can read that column without issues:

1 Like

What we have is MapR 6 with arcengine 2.9.3.

@Dao-I_Tony_Lin okay, if you upgrade to Arcadia 4.5 and use ORC storage format, DATE field types can be read from within Arcadia.