Semantic Model: kyvos.split.combination

Description:

This property specifies how to handle small file splits of the data source. Small files refer to files smaller than the DFS block size. This allows them to be handled by a single mapper instead of using one for each file.

Values and behavior:

• true - Combines smaller files into a single file.

• false - Doesn't combine any files.

Default value:

false

Scope:

• Connection: If the property is set at the connection, then the property value is applicable for all semantic model process jobs.

• Semantic Model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model's process job.

Comes into effect:

The value of the property can be changed at any time and will be respected in the next process instance.

Dependencies and Related Properties:

• Value for kyvos.build.execution.engine property must be set as MapReduce

Recommendation:

This property should be set to True when input data consists of a large number of small files. By combining small files, mappers will operate more efficiently, and the total number of mappers required is reduced, in turn reducing the process time.