Kyvos now comes with an in-house data storage layer called the Kyvos Analytical Store. This layer provides efficient data storage and retrieval capabilities that help to optimize both cost and time savings. The Kyvos Analytical Store is specifically designed to support complex, long-running queries that process substantial portions of the stored dataset. It is equipped with a columnar-vectorized query execution engine that processes a large batch of values, referred to as a "vector", in a single operation. This vectorized query execution significantly improves performance, especially in OLAP queries.
Note
When using semantic models with Kyvos Analytical Store, you must process the full data of the semantic model if it has not been processed before. Additionally, it is recommended to avoid using a previously processed semantic model after enabling the Kyvos Analytical Store. This is because executing queries on a previously processed semantic model may slow down as a result of Kyvos Analytical Store memory calculations.
Certified Providers and supported Operating Systems for Kyvos Analytical Store
The Kyvos 2024.1.x is certified to work on the following provider and OS versions with the Kyvos Analytical Store.
Provider | OS Version |
|---|---|
CDP 7.1.8/CDH-6.3.2 | Ubuntu 18.04 |
AWS | Centos 7.9 and Amazon Linux 2 |
Azure | RHEL 8.6 and RHEL7.8 |
GCP | CentOS 7.9, RHEL 8.8, and Ubuntu 20.04 |
Prerequisites
To use Kyvos Analytical Store, you must ensure the following:
Set the value of ENABLE_KYVOS_ANALYTICAL_STORE property as Yes.
Set the value of ELASTIC_DEPLOYMENT as Disable.
Set the value of kyvos.cuboid.replication.type property as Local on the semantic model.
Adding Kyvos Analytical Store Jar in classpath
To add Kyvos Analytical Store Jar in classpath, perform the following steps.
Login to Kyvos and navigate to the Connections page.
On the Connections page, click Properties.
On the Edit Properties dialog box, search “kyvos.connection.classpath.common.jar” and enter the ../lib/duckdb_jdbc.jar value in the existing class path.
Restart Kyvos services.
Using Kyvos Analytical Store for semantic models
To use the Kyvos Analytical store as data storage layer for a semantic model, use the following property on the semantic model.
Property: kyvos.build.aggregate.type
Value: Tabular
Scope: Semantic model advanced property
Kyvos Analytical Store memory calculations
Parameter | Memory Calculation |
|---|---|
HEAP_MEMORY | VM_TOTAL_MEMORY / 4 |
OFF_HEAP_MEMORY | VM_TOTAL_MEMORY / 8 |
KYVOS_ANALYTICAL_STORE_MEMORY | VM_TOTAL_MEMORY / 4 |
For example, If you have a 64 GB Query Engine Virtual Machine, the memory calculations are as follows:
HEAP_MEMORY = 16 GB
OFF_HEAP_MEMORY= 8 GB
KYVOS_ANALYTICAL_STORE_MEMORY= 16 GB
Memory tuning parameters
The Heap, Off-heap, and other required memory on Query Engines are automatically configured if memory mode is set to Auto mode.
For more details, see the Kyvos Java options page in Kyvos Manager.
To enable Kyvos Analytical store, perform the following steps.
Steps for Databricks
You must change Spark configurations to use managed disk. Ensure that you must not change the configuration in the default root (/tmp) volume.
To change Spark configurations to use managed disk, perform the following steps.
Databricks > Advance configuration > Spark
In the Spark Configurations, add the spark.local.dir /local_disk0 property where the local_disk0 is the managed disk.
This can be verified using the df -h command from a notebook
Add the SPARK_WORKER_DIR=/local_disk0 value in the Environment variables.