Description:
This property specifies the source for fetching the metadata details of data having a dynamic schema where the structure is different between the file and the HCatalog definition.
Values and behavior:
Table - Metadata will come from the HCatalog metastore.
File - Metadata will come from the Parquet file.
Default value:
Table
Scope:
Dataset: If the property is set for a particular dataset, then the value will be applicable for a particular dataset and will override the value applied (if any) at the connection.
Comes into effect:
The value of the property can be changed at any time. If the schema changes, then the dataset must be saved again with the updated field lists to use them further.
Dependencies and related properties:
The dataset must be built on a dynamic schema from a Parquet file from the HCatalog table.
Recommendation:
Generally, the metadata should come from HCatalog as HCatalog can handle dynamic schema. This property should only be changed when the file has a proper structure and HCatalog does not.