No- Spark model processing through Kubernetes and Query Engine

Deploy Kyvos through Kubernetes and Query Engine

You can use no Spark model processing to deploy Kyvos through Query Engine or Kubernetes for Azure and GCP.

• AWS: You can use no Spark model processing to deploy Kyvos through Query Engine.

• Azure: For Kubernetes, Kyvos processes the semantic model using Azure’s managed service AKS (Azure Kubernetes Service). The Azure cluster is deployed via ARM templates. You can create a cluster without Spark or process the semantic model utilizing Spark Mode within ARM templates.

  • Prerequisites to deploy Kyvos using Kubernetes

  • Creating ARM Template for Automated Azure Deployment

• GCP: For Kubernetes, Kyvos processes the semantic model using Google Cloud's managed service GKE (Google Kubernetes Engine). The GKE cluster is deployed through GCP Installation Files. Using the scripts, you can select a No-Spark-based cluster or process the semantic model using Spark Mode.
  To proceed with a no-spark-based deployment mode, you must use a Dataproc cluster (either new or existing).
  For more details about how to deploy Kyvos on GCP using the no-Spark model, see the following section:

  • Prerequisites to deploy Kyvos using Kubernetes

  • Creating Kyvos Resources for GCP using Script

Post deployment steps for all clouds (AWS, Azure and GCP)

After deploying Kyvos using no-Spark processing model, perform the following post deployment steps.

1. Add the kyvos.connection.readUsingCustomFS.jobs.internal=NONE property from the Kyvos connections page

2. Modify the values of the following properties in the advance properties of semantic model job:

   • kyvos.process.compute.type=KYVOS_COMPUTE

   • kyvos.build.aggregate.type=TABULAR

3. Set the kyvos.process.datastore.properties property on Hadoop Connection properties and set the value as SET disabled_optimizers = 'join_order';SET memory_limit='40GB';SET threads TO 1;

4. Restart Kyvos services.

Post deployment steps for Azure

The post-deployment steps must be executed since the Data Lake connection is blank on the Azure no Spark cluster.

1. SSH Kyvos Manager from terminal.

2. Navigate to /data/kyvos/app/kyvos/olapengine/conf/

3. In the providerInfo.xml file, add Hadoop cluster for Azure provider list, including all necessary details.

<DEPLOYMENT_PROVIDER_MAPPING PROVIDERLIST="5">
    <TYPE NAME="AZURE"></TYPE>
</DEPLOYMENT_PROVIDER_MAPPING>
<PROVIDERLIST NAME="5">
    <ALLOWED_PROVIDERS><![CDATA[HADOOP_CLUSTER,LOCAL_PROCESS_CLUSTER,DATABRICKS,PRESTO,ATHENA,POSTGRESQL,AZURESQLDB,GENERIC,SNOWFLAKE]]></ALLOWED_PROVIDERS>
</PROVIDERLIST>

2. Update the snapshot Bundle for Kyvos BI server configurations.

3. Restart the services.

Important points to know

To process semantic model without spark, you must do the following:

• From the Kyvos Connection page, do the following:

  • For ROLAP queries, set the kyvos.connection.defaultsqlengine property’s value as True.

  • Add the kyvos.connection.isRead property and set is value as True.

• Modify the value of the following semantic model advanced properties on Kyvos Web Portal:

  • kyvos.sqlwarehouse.catalog = ‘Your catalog name’ 

  • kyvos.sqlwarehouse.tempdb = ‘Your Database name’ 

  • kyvos.build.aggregate.type = TABULAR 

  • Kyvos.process.compute.type = Kyvos Compute 

• You can also set subtype to select no Spark model processing for semantic model via property.
  To do this, navigate to the Kyvos Properties page and update the KYVOS_PROCESS_COMPUTE_SUBTYPE property and restart Kyvos services.