Kyvos has introduced an in-house data storage layer called the Kyvos Analytical Store. The data store is designed to support complex, long-running queries that process significant portions of the stored dataset.
This layer provides efficient data storage and retrieval capabilities that help to optimize both cost and time savings. The Kyvos Analytical Store is specifically designed to support complex, long-running queries that process substantial portions of the stored dataset. It is equipped with a columnar-vectorized query execution engine that processes a large batch of values, referred to as a "vector", in a single operation. This vectorized query execution significantly improves performance, especially in OLAP queries.
Note
To use semantic models with Kyvos Analytical store, you must process the full data of the semantic model if it has already been processed before. Additionally, it is recommended not to use the semantic model that was previously processed if you have enabled Kyvos Analytical Store. After enabling Kyvos Analytical Store, the previously processed semantic model may slow down the query execution time due to the new memory configuration.
Certified Providers and supported Operating Systems for Kyvos Analytical Store
The Kyvos 2024.1.x is certified to work on the following provider and OS versions with the Kyvos Analytical Store.
Provider | OS Version |
|---|---|
CDP 7.1.8/CDH-6.3.2 | Ubuntu 18.04 |
AWS | Centos 7.9 and Amazon Linux 2 |
Azure | RHEL 8.6 and RHEL7.8 |
GCP | CentOS 7.9, RHEL 8.8, and Ubuntu 20.04 |
Prerequisites
To use Kyvos Analytical Store, you must ensure the following:
Add Kyvos Analytical Store Jar in classpath
The value of ENABLE_KYVOS_ANALYTICAL_STORE property should be set as Yes.
The value of ELASTIC_DEPLOYMENT should be set as Disable.
The value of kyvos.cuboid.replication.type property should be set as Local on the semantic model.
Add Kyvos Analytical Store Jar in classpath
To add Kyvos Analytical Store Jar in classpath, perform the following steps.
Login to Kyvos and navigate to the Connections page.
On the Connections page, click Properties.
On the Edit Properties dialog box, search “kyvos.connection.classpath.common.jar” and enter the ../lib/duckdb_jdbc.jar value in the existing class path.
Restart Kyvos services.
Using Kyvos Analytical Store for semantic models
To use the Kyvos Analytical store as data storage layer for a semantic model, use the following property on the semantic model.
Property: kyvos.build.aggregate.type
Value: Tabular
Scope: Semantic model advanced property
Kyvos Analytical Store memory calculations
Parameter | Memory Calculations |
|---|---|
HEAP_MEMORY | VM_TOTAL_MEMORY / 4 |
OFF_HEAP_MEMORY | VM_TOTAL_MEMORY / 8 |
KYVOS_ANALYTICAL_STORE_MEMORY | VM_TOTAL_MEMORY / 4 |
For example, If you have 64 GB QE VM, the following will be the memory calculations:
HEAP_MEMORY = 16 GB
OFF_HEAP_MEMORY= 8 GB
KYVOS_ANALYTICAL_STORE_MEMORY= 16 GB
Memory tuning parameters
The Heap, Off-heap, and other required memory on Query Engines are automatically configured if memory mode is set to Auto mode.
For more details, see the Kyvos Java options page in Kyvos Manager.
To enable Kyvos Analytical store, perform the following steps.