Description:
This property specifies the size of the blocks or splits used by a job launched by Kyvos. This value controls the number of Mappers required for process jobs. It also controls the size of cuboids and the number of cuboids when applied to a semantic model.
Values and behavior:
Any positive number value representing the split size in bytes
Default Value:
Defaults to the Hadoop configuration in the HDFS-site.xml file.
Scope:
- Connection: If the property is set at the connection, then the property value is applicable for all transformation build, semantic model processes, or data profile jobs launched using Spark.
- Semantic model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model’s job.
Comes into effect:
This property comes into effect only when the execution engine is set as MapReduce. The value of the property can be changed at any time and will be respected in the next build instance.
Dependencies and related properties:
- mapreduce.input.fileinputformat.split.maxsize
Recommendation:
This is a balanced property. A higher value will result in fewer, but larger cuboids, whereas a lower value will result in more, but smaller cuboids.