Kyvos has introduced now comes with an in-house data storage layer called the Kyvos Analytical Store. The data store is designed to support complex, long-running queries that process significant portions of the stored dataset. This layer provides efficient data storage and retrieval capabilities that help to optimize both cost and time savings. The Kyvos Analytical Store is specifically designed to support complex, long-running queries that process substantial portions of the stored dataset. It is equipped with a columnar-vectorized query execution engine that processes a large batch of values, referred to as a "vector", in a single operation. This vectorized query execution significantly improves performance, especially in OLAP queries.
...
The Kyvos 2024.1.x is certified to work on the following provider and OS versions with the Kyvos Analytical Store.
Provider | OS Version |
|---|
CDP 7.1.8/CDH-6.3.2 | Ubuntu 18.04 |
AWS | Centos 7.9 and Amazon Linux 2 |
Azure | RHEL 8.6 and RHEL7.8 |
GCP | CentOS 7.9, RHEL 8.8, and Ubuntu 20.04 |
Prerequisites
To use Kyvos Analytical Store, you must ensure the following:
Add Kyvos Analytical Store Jar in classpath.
...
Set the value of ENABLE_KYVOS_ANALYTICAL_STORE property
...
as Yes.
...
Set the value of ELASTIC_DEPLOYMENT
...
as Disable.
...
Set the value of kyvos.cuboid.replication.type property
...
as Local on the semantic model.
Adding Kyvos Analytical Store Jar in classpath To add Kyvos Analytical Store Jar in classpath, perform the following steps.
...
| Aura tab collection |
|---|
| params | JTdCJTIyZ2VuZXJhbCUyMiUzQSU3QiUyMnRhYlNwYWNpbmclMjIlM0EwJTJDJTIydGFiV2lkdGglMjIlM0ExMDAlMkMlMjJ0YWJIZWlnaHQlMjIlM0E1MCUyQyUyMmRpcmVjdGlvbiUyMiUzQSUyMmhvcml6b250YWwlMjIlN0QlMkMlMjJjb250ZW50JTIyJTNBJTdCJTIyYmFja2dyb3VuZENvbG9yJTIyJTNBJTdCJTIyY29sb3IlMjIlM0ElMjIlMjNmZmYlMjIlN0QlMkMlMjJib3JkZXIlMjIlM0ElN0IlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyd2lkdGglMjIlM0ExJTJDJTIydG9wJTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmNvbG9yJTIyJTNBJTdCJTIybGlnaHQlMjIlM0ElMjIlMjNjY2NlY2YlMjIlN0QlN0QlMkMlMjJwYWRkaW5nJTIyJTNBJTdCJTIydG9wJTIyJTNBMTAlMkMlMjJyaWdodCUyMiUzQTEwJTJDJTIyYm90dG9tJTIyJTNBMTAlMkMlMjJsZWZ0JTIyJTNBMTAlN0QlN0QlMkMlMjJhY3RpdmUlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzZjU4MjI3JTIyJTdEJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzMDAwMDAwJTIyJTdEJTJDJTIydGV4dEFsaWduJTIyJTNBJTIybGVmdCUyMiUyQyUyMmZvbnRXZWlnaHQlMjIlM0ElMjJib2xkJTIyJTdEJTdEJTJDJTIyaG92ZXIlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSUyMiUyM2RmZTFlNiUyMiU3RCUyQyUyMnRleHQlMjIlM0ElN0IlMjJmb250U2l6ZSUyMiUzQTE4JTJDJTIyY29sb3IlMjIlM0ElMjIlMjM1ZTZjODQlMjIlMkMlMjJ0ZXh0QWxpZ24lMjIlM0ElMjJsZWZ0JTIyJTJDJTIyZm9udFdlaWdodCUyMiUzQSUyMmxpZ2h0ZXIlMjIlN0QlN0QlMkMlMjJpbmFjdGl2ZSUyMiUzQSU3QiUyMmJhY2tncm91bmRDb2xvciUyMiUzQSU3QiUyMmNvbG9yJTIyJTNBJTIyJTIzZjRmNWY3JTIyJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSUyMiUyMzVlNmM4NCUyMiUyQyUyMnRleHRBbGlnbiUyMiUzQSUyMmxlZnQlMjIlMkMlMjJmb250V2VpZ2h0JTIyJTNBJTIybGlnaHRlciUyMiU3RCUyQyUyMmJvcmRlciUyMiUzQSU3QiUyMnRvcCUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJ3aWR0aCUyMiUzQTElMkMlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyY29sb3IlMjIlM0ElN0IlMjJsaWdodCUyMiUzQSUyMiUyM2NjY2VjZiUyMiU3RCU3RCU3RCU3RA== |
|---|
|
| Aura tab |
|---|
| summary | Kyvos versions up to 2024.1 |
|---|
| params | JTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHZlcnNpb25zJTIwdXAlMjB0byUyMDIwMjQuMSUyMiU3RA== |
|---|
|
Login to Kyvos Manager and navigate to Properties > queryengine.properties. In the queryengine.properties, search for the DS_CONNECTION_CONF property. Restart the Query Engine services from the Kyvos Manager dashboard.
For memory configuration, perform the following post-deployment steps. Open the Kyvos Java options page in Kyvos Manager and navigate to the Query Engine section. If Configure Heap memory for Query Engine is set as Auto, no change is required. If Configure Heap memory for Query Engine is set as Manual, reduce Off-Heap memory by 10 GB.
For Example, Suppose the total machine memory is 128 GB, whereas 45 GB is assigned to Heap, and 66 GB is assigned to Off-Heap. In that case, you need to set Heap as 45 GB and Off-Heap as 56 GB so Kyvos can use the remaining 10 GB internally for the new data store-based architecture.
| Aura tab |
|---|
| summary | Kyvos 2024.1.1 onwards |
|---|
| params | JTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMDIwMjQuMS4xJTIwb253YXJkcyUyMiU3RA== |
|---|
|
| Panel |
|---|
| panelIconId | atlassian-note |
|---|
| panelIcon | :note: |
|---|
| bgColor | #DEEBFF |
|---|
| Before enabling Kyvos Analytical Store, ensure that Query Engine cluster is running with the required (Full, high, moderate, or Low) capacity, and disable cluster scaling. For this, set the value of ELASTIC_DEPLOYMENT should be set as Disable. |
Login to Kyvos Manager and navigate to Properties > global.properties. Set the value of ENABLE_KYVOS_ANALYTICAL_STORE property as Yes. Restart all Kyvos services (BI Server, Query Engine, and Kyvos Web Portal) from the Kyvos Manager dashboard. Open the Kyvos Java options page in Kyvos Manager and navigate to the Query Engine section, and ensure that the Configure Heap memory for Query Engine is set as Auto. Kyvos will now automatically configure memory used for the Analytical Store.
|
Steps for Databricks
You must change Spark configurations to use managed disk. Ensure that you must not change the configuration in the default root (/tmp) volume.
To change Spark configurations to use managed disk, perform the following steps.
Databricks > Advance configuration > Spark
Add the SPARK_WORKER_DIR=/local_disk0 value in the Environment variables.
