Description:
This property specifies the amount of off-heap memory per executor when jobs are executed using Spark. Kyvos delegates this property value to the same-named property of Spark when Kyvos submits a transformation process, semantic model process, or data profile job to Spark.
Values and behavior:
Any positive integer (represents value in MB).
For example: 6144
Default value:
6144: This means 6144 MB of memory will be allocated to each Spark executor as the off-heap memory.
Scope:
- Connection: If the property is set at the connection, then the property value is applicable for all transformation processes, semantic model processes, or data profile jobs launched using Spark.
- Semantic model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model’s process.
Comes into effect:
This property comes into effect only when kyvos.build.execution.engine is set as Spark. This means when the underlying build engine is set as Spark. The value of the property can be changed at any time and will be respected in the next process.
Dependencies and related properties:
- kyvos.build.execution.engine
- spark.executor.memory
- kyvos.build.execution.engine
- spark.dynamicAllocation.minExecutors
- spark.dynamicAllocation.maxExecutors
- Semantic Model: spark.executor.cores
Recommendation:
The default value of the property should work fine for most scenarios. If the semantic model build jobs fail due to physical memory-related failures, then the job memory can be increased using this property. The values are recommended to be in the range of 4096 to 8192.