Semantic Model: kyvos.records.skipbadrows

Description:
This property specifies how to handle invalid rows when reading data during a dataset or semantic model process job. The rows that have any mismatch in the data type or in the number of columns (specifically in case of files created on HCatalog) are considered invalid rows.

Values and behavior:

True: Kyvos will skip the invalid data rows while reading data during the process process. The process will fail only if the number of skipped records exceeds the threshold defined in the Semantic Model: kyvos.records.skipbadrows.threshold property. In this case, the processed row count as shown in the process summary may be lesser than the total number of rows in the source data.
False: Kyvos will not skip any invalid row and immediately fail the process as soon as an invalid record is encountered.

Default Value:
True

Scope:

Connection: If the property is set at the connection, then the property value is applicable for all dataset process, semantic model process, or data profile jobs.
Semantic model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model’s process job.

NOTE: If the property is set on a dataset and a dataset is processed, then the value will override the connection level value for that dataset process job.

Comes into effect:
The value of the property can be changed at any time (not recommended) and comes into effect in the next process.

Dependencies and Related Properties:

Recommendation:
If data is likely to have invalid rows that can be skipped, then set this property to true with an appropriate threshold. We do not recommend changing the value of this property in between processes, as a change may result in inconsistent aggregated values.