Semantic Model: spark.dynamicAllocation.minExecutors

Description:
This property specifies the minimum number of executors for the Spark jobs in the semantic model process. Kyvos delegates this property value to the same-named property of Spark when Kyvos submits a dataset process, semantic model process, or data profile job to Spark.

Values and behavior:
Any positive integer.

Default value:
0 - Enables Spark's intelligence to identify the minimum number of executors, any value greater than 0 mandates Spark to start the job only when available memory is sufficient enough to launch these many executors

Scope:

• Connection: If the property is set at the connection, then the property value is applicable for all semantic model processes or data profile jobs launched using Spark.

• Semantic model: If the property is set on a semantic model, then the value will override the connection level value for that semantic model’s process job.

Comes into effect:
This property comes into effect only when kyvos.build.execution.engine is set as Spark. The value of the property can be changed at any time and will be respected in the next semantic model job.

Dependencies and related properties:

• kyvos.build.execution.engine

• spark.dynamicAllocation.maxExecutors

• spark.executor.cores

• spark.executor.memory

• Semantic Model: spark.yarn.executor.memoryOverhead