Description:
This property specifies the number of invalid rows that can be skipped with the Semantic Model: kyvos.records.skipbadrows property. Once this threshold is reached, the dataset or semantic model build will fail.
Values and behavior:
Any positive number. The process will continue till the time when skipped record count is lesser than or equal to the specified value, else will fail.
When set to 0, there is no limit to the number of rows that may be skipped.
Default value:
100 - Once 100 invalid rows are found, the process will fail.
Scope:
Connection: If the property is set at the connection, then the property value is applicable for all semantic model processes, or data profile jobs launched using Spark.
Semantic model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model’s process job.
Comes into effect:
This property comes into effect only when Semantic Model: kyvos.records.skipbadrows is set to TRUE. The value of the property can be changed at any time and will be respected in the next build.
Dependencies and related properties:
kyvos.records.skipbad.rows must be set to TRUE.
Recommendations:
The threshold is applied per MapReduce Mapper or Spark Map task. For a build that has a threshold of 10,000 and 10 Mapper tasks, the build will not fail until a single task encounters 10,000 invalid rows. Each of the tasks can have 9,000 invalid rows, which means 90,000 invalid rows in total; and the build will succeed.