Semantic Model: spark.executor.cores

Description:
This property specifies the number of cores each Spark executor can consume when Kyvos submits a request to Spark. Kyvos delegates this property value to the same-named property of Spark when Kyvos submits a dataset process, semantic model process, or data profile job to Spark.

Values and behavior:
Any positive integer value. Spark will use the specified number of cores for each executor.

Default value:
4

Scope:

• Connection: If the property is set at the connection, then the property value is applicable for all dataset processes, semantic model processes or data profile jobs launched using Spark.

• Semantic model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model’s job.

Comes into effect:
This property comes into effect only when the underlying process engine is set to Spark. The value of the property can be changed at any time and will be respected in the next process.

Dependencies and related properties:

• kyvos.build.execution.engine

• Semantic Model: spark.dynamicAllocation.minExecutors

• spark.dynamicAllocation.maxExecutors

• spark.dynamicAllocation.initialExecutors

Recommendation:
Either 4 or 5 cores are recommended to be used. The recommended is 4 as per our lab tests.