Page Comparison

Applies to: Kyvos Enterprise  Kyvos Cloud Kyvos Cloud (Managed Services SaaS on AWS) Kyvos Azure AWS Marketplace

Kyvos AWS Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI) Kyvos Free ( Limited offering for AWS)

...

1. Configure Cluster and Query Engine Scheduling to save cost and use cloud resources only when needed. 

   1. You can create a schedule to: 

   2. Shutdown cluster for any time interval 

   3. Start cluster for any time interval. 

   4. Schedule Query Engines for any time interval 

2. Auto Scaling and Auto Termination Policy on Databricks Cluster to save cost and achieve high cluster Utilization. 

   1. Auto Termination: You can also set auto termination for a cluster. During cluster creation, you can specify an inactivity period in minutes after which you want the cluster to terminate. If the difference between the current time and the last command run on the cluster is more than the inactivity period specified, Databricks automatically terminates that cluster. 
      A cluster is considered inactive when all commands on the cluster, including Spark jobs, Structured Streaming, and JDBC calls, have finished executing. This does not include commands run by SSH-ing into the cluster and running bash commands. 
      Standard clusters are configured to terminate automatically after 120 minutes. You can modify the default value as needed. 

   2. Auto Scaling: When you create a Databricks cluster, you can either provide a fixed number of workers for the cluster or provide a minimum and maximum number of workers for the cluster.  
      When you provide a fixed size cluster, Databricks ensures that your cluster has the specified number of workers. When you provide a range for the number of workers, Databricks chooses the appropriate number of workers required to run your job. This is referred to as autoscaling. 
      With autoscaling, Databricks dynamically reallocates workers to account for the characteristics of your job. Certain parts of your pipeline may be more computationally demanding than others, and Databricks automatically adds additional workers during these phases of your job (and removes them when they’re no longer needed).  
      Autoscaling makes it easier to achieve high cluster utilization because you don’t need to provision the cluster to match a workload. This applies especially to workloads whose requirements change over time, but it can also apply to a one-time shorter workload whose provisioning requirements are unknown.
      Autoscaling thus offers two advantages: 

...