...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Applies to: 
Kyvos Enterprise Kyvos Azure Marketplace
Kyvos AWS Marketplace Kyvos Free (
...
Prerequisites for semantic model process size and time optimization
...
- kyvos.build.spark.levelJob.tasks: Use this property to configure the number of tasks for reducing stage of Level1 and Level_DistCount jobs for Full or/and Incremental semantic model process job when executing through Spark engine. You can set this property to any positive integer and thereafter that number of tasks will get launched for the reducer stage to increase parallel tasks if the required resources are available on the cluster.
Default value is automatic based on data loads. - spark.dynamicAllocation.minExecutors : Use this spark property sets the lower bound for the number of executors if the dynamic allocation is enabled. See details.
- spark.yarn.executor.memoryOverhead: Use this spark property to set the amount of off-heap memory (in megabytes) to be allocated per executor. This memory accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%). See details.
- spark.driver.memory: Use this property to set the amount of memory to be used for the driver process, i.e., where SparkContext is initialized. ( e.g., 1g, 2g).
- kyvos.spark.executor.memory.level1: Use this property t o set the spark executor memory for level1 job(s) launched during the full and/or incremental semantic model process.
- kyvos.spark.executor.cores.level1: Use this property to set the spark executor cores for level1 job(s) launched during the full and/or incremental semantic model process.
- spark.executor.memory: Use this property to set the amount of memory to be used per executor process ( e.g., 2g, 8g). See details.
- spark.executor.cores: Use this property to set the number of cores to be used on each executor. This property is for YARN and standalone mode only in spark. In standalone mode, setting this parameter allows an application to run multiple executors on the same worker if there are enough cores on that worker. Otherwise, only one executor per application will run on each worker. See details.
- spark.dynamicAllocation.maxExecutors: Use this property to set the upper bound for the number of executors if the dynamic allocation is enabled. See details.
| Info |
|---|
For Azure Databricks based environment you need explicitly define/modify the spark properties in Databricks Advance Spark Configuration. |
...