Sliding window

Applies to: Kyvos Enterprise  Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

A sliding window refers to how much historical data should be retained within the semantic model (in years, months, or days, etc.) For example, if a semantic model is incrementally processed daily, then a sliding window of 10 days means that the semantic model will only retain data for the last 10+1 days.

To set up a sliding window, you must define partitions for the semantic model. 

• Use a Time dimension in the semantic model to set up the sliding window. Note that larger numbers of partitions can affect performance. 

• When you update or replace data in a semantic model partition, you must specify the full range of data for that partition. For example, if a semantic model partition contains a day’s worth of data, then you must specify the time range as a day.

• Using a sliding window, you can define the range for which you want to keep the data and delete the older data. For example, with a partition, you can keep one year's worth of data, while using a sliding window, you can maintain it for five years.

• If you choose to keep data for the latest five years in the semantic model, data (and corresponding metadata) that is out of the sliding window range will be deleted. For example, if new data comes in with an incremental process. The semantic model job Summary shows the information of deleted data.

• Only the valid members, according to sliding window duration, will be visible for all metadata queries across all viewer interfaces, i.e., MDX, SQL, Kyvos visualization, and third-party tools. There will be no impact on the row-level/column-level security filters.

• You can update the range of an existing sliding window without needing to reprocess the semantic model. After you update the range of a sliding window, when you incrementally add data, the system will automatically drop old data.

• You can use sub-partitions to fine-tune the data updated in your sliding window.

Let us consider a semantic model without sub-partitions. Now if you have set the semantic model partition on month, then partitions are created for January, February, March, and so on. Now, if an incremental data process brings in updated data for all days of January, the complete partition will be replaced with the updated data. However, if data is updated only for a specific date, say 15th January, then in this case, the incremental process with Replace Partition will update data only for 15th January and delete the data for the rest of the days of January.

To avoid such scenarios, it is recommended to use sub-partitions. 

Considering the example above, you can define the partition on the month and the sub-partition on the day. This will ensure only data for the specific day will be updated, and the rest will remain the same.

To create a sliding window, perform the following steps.

1. From the Toolbox area, select Semantic Models.

2. Select a semantic model that includes time data and click Refine.

3. Click the Partition Strategy tab.

4. For Base Partition On, browse the data and choose from the available time-based options.
   Or, click Search to find the option you want to use quickly.

5. Set the amount of data each partition should have.
   For example, if the partition is based on days, set how many days each partition contains.

6. Optionally, click Sub Partitions to set up sub-partitions.
   Click the list box and browse to select the sub-partition you want to add.

   • Optionally, click the plus button to add an additional sub-partition or the minus button to remove one.

7. Click the checkbox to enable a sliding window.
   Set the number of partitions you want to keep the data for a set.

   • For example, if you based the partition on days, you can choose to set the window size to 10 days.

   • The dialog box alerts you how much data is added to the semantic model before the oldest partition is deleted.

8. Click Save.
   
   

   Open

   

Related topic

• Partition strategy