Applies to: (tick) Kyvos Enterprise  (tick) Kyvos Cloud (SaaS on AWS) (tick) Kyvos AWS Marketplace

(tick) Kyvos Azure Marketplace   (tick) Kyvos GCP Marketplace (tick) Kyvos Single Node Installation (Kyvos SNI)


You can set up the schedule for all of the process types. Schedule the frequency and the time when semantic models get processed. You can also set semantic model process properties. To learn about reverting to or managing backup processes, see Managing semantic model job instances.

Note

  • From Kyvos 2024.11 onwards, you can use the semantic model in a tabular way.

  • From Kyvos Manager, you can set the duration (in hours) to limit the semantic model process time using the BUILD_TIME_THRESHOLD (OlapEngine.property). Once the semantic model process limit is reached, the process will fail. If you resume the semantic model process, it will use the same duration that you set during the initial semantic model process. 

There are two options for scheduling the different job types. See Semantic model process types. Click the i icon next to the job type list to get a description of each process type. 

  1. From the Toolbox, choose Semantic Model.

  2. Select a semantic model from the list and click Process

  3. From the top right of the window, click Schedule Job.

  4. From the Job Type list, select the Process Full Data option.

  5. Select the job connection that you want to use. In case there are multiple job connections in the underlying registered files in the semantic model, semantic model validation will be triggered on the default Hadoop connection.

  6. You can review and alter the design using recommendations.

  7. If the last job failed, you can resume from the last failure state. Click Resume to confirm. Use the job Source Datasets options to specify whether to materialize datasets with every job or not.

  8. Depending upon the type of Aggregation Strategy selected in the semantic model design, you can add additional query patterns or analyze query patterns using the corresponding links. 

    1. Optionally, click filter queries to select the queries to be used for the aggregation strategy.

    2. The system will automatically generate aggregates definitions based on the latest data profiles, all available historical queries, and additional query patterns.

  9. Select Process as Metadata and Data or Metadata only. This is applicable only when Kyvos compute-based processing is enabled on the semantic model.

    • Metadata and Data: Selecting this option will populate both the dimension metadata cache and data (Kyvos Analytical Store). Queries will be served from Kyvos Analytical Store.

    • Metadata Only: The system will populate only dimension metadata cache. Your queries will be submitted directly to the underlying data source.

  10. Click one of the following options to select your materialization strategy.
    NOTE: This is applicable only when Kyvos compute and raw data querying is enabled on the semantic model. While scheduling Process Full Data for a semantic model, the Both option must be selected.

    1. Aggregated Data: The system will keep the aggregated data (and update it in subsequent semantic model processing as per the query patterns) in Kyvos Analytical store. This data will be used to serve the aggregates queries.

    2. Raw Data: The system will keep the raw data in Kyvos Analytical store. This data will be used to serve those queries for which smart aggregates are not available in the Kyvos Analytical store.

    3. Both: The system will keep aggregated and raw data both.

  11. Choose to run once, now, or set up a recurring schedule. With each option, you can also view or change the properties currently in use.

    • If you choose Once, select a date, time, and time zone.

    • To set up a recurring process, enter the start and end times, the frequency, date, days, week, moths, hours, minutes, and time zone.
      Specify whether to job once per day or repeat. Kyvos picks the time zone from where the report server is running, as the default time zone. You can also specify configure semantic model job schedules for specific days of the week using a recurring schedule. This includes flexible options to schedule jobs on non-consecutive days, such as alternating days, specific weekdays, or any custom combination of days.

  12. Click Schedule to start the process.

image-20241125-185558.png

  1. From the Toolbox, choose Semantic Model.

  2. Select a semantic model from the list and click Process.

  3. From the top right of the window, click Schedule Job.

  4. From the Job Type list, select the Process Incremental Data option.

  5. Select the Job connection that you want to use. In case there are there are multiple job connections in the underlying datasets in the semantic model, semantic model validation will be triggered on the default Hadoop connection.

  6. From the Process list, select any of the following:

    1. All Newer Data: Detects and processes all the new data in the files used in semantic model design.

    2. Only Fact Adjustment: Detects and processes only adjustable measures in fact files used in semantic model design. In this case, the old data is not overridden, and the adjustment data can be incrementally loaded in the semantic model without touching any previous data. The adjustments will be made at the lowest granularity of the measure data. Adjustment job can be completed quickly and there can be multiple adjustments in a day. The adjustment details are also displayed in the job summary.

  7. Use the Replace Partition option to update the already processed data. This option drops out old data and adds current data - keeping the partition size constant. See Replace Partitions.

  8. Depending upon the type of Aggregation Strategy selected in the semantic model design, you can select any of the following:

    1. Generate additional aggregates, keeping the existing ones (if any): Additional aggregates definition recommendation will be taken keeping existing aggregates as it is.

    2. Use existing aggregates definition: No new aggregates definition recommendation will be taken, aggregates will be created using the existing definition only.

  9. Add additional query patterns or analyze query patterns using the corresponding links. 

  10. Optionally, click filter queries to select the queries to be used for the aggregation strategy.

  11. Select Process as Metadata and Data or Metadata only. This is applicable only when Kyvos compute-based processing is enabled on the semantic model.

    • Metadata and Data: Selecting this option will populate both the dimension metadata cache and data (Kyvos Analytical Store). Queries will be served from Kyvos Analytical Store.

    • Metadata Only: The system will populate only dimension metadata cache. Your queries will be submitted directly to the underlying data source.

  12. Click one of the following options to select your materialization strategy.
    NOTE: This is applicable only when Kyvos compute and raw data querying is enabled on the semantic model. While scheduling Process Full Data for a semantic model, the Both option must be selected.

    1. Aggregated Data: The system will keep the aggregated data (and update it in subsequent semantic model processing as per the query patterns) in Kyvos Analytical store. This data will be used to serve the aggregates queries.

    2. Raw Data: The system will keep the raw data in Kyvos Analytical store. This data will be used to serve those queries for which smart aggregates are not available in the Kyvos Analytical store.

    3. Both: The system will keep aggregated and raw data both.

  13. Choose to run once, now, or set up a recurring schedule. With each option, you can also view or change the properties currently in use.

    • If you choose Once, select a date, time, and time zone.

    • To set up a recurring process, enter the start and end times, the frequency, date, days, week, moths, hours, minutes, and time zone.
      Specify whether to job once per day or repeat. Kyvos picks the time zone from where the report server is running, as the default time zone. You can also specify configure semantic model job schedules for specific days of the week using a recurring schedule. This includes flexible options to schedule jobs on non-consecutive days, such as alternating days, specific weekdays, or any custom combination of days.

    • Click Schedule to start the process.

image-20241125-185716.png
  1. From the Toolbox, choose Semantic Model.

  2. Select a semantic model from the list and click Process

  3. From the top right of the window, click Schedule Job.

  4. From the Job Type list, select the Process Test Data option.

  5. Select the Job connection that you want to use. In case there are there are multiple process connections in the underlying registered files in the semantic model, semantic model validation will be triggered on the default Hadoop connection.

  6. From Input Data, you can specify the number of last fact data partitions or the number of records to be processed for the test process.

    Use the Process Source Datasets options to specify whether to materialize datasets with every process or not.

  7. Select Process as Metadata and Data or Metadata only. This is applicable only when Kyvos compute-based processing is enabled on the semantic model.

    • Metadata and Data: Selecting this option will populate both the dimension metadata cache and data (Kyvos Analytical Store). Queries will be served from Kyvos Analytical Store.

    • Metadata Only: The system will populate only dimension metadata cache. Your queries will be submitted directly to the underlying data source.

  8. Click one of the following options to select your materialization strategy.
    NOTE: This is applicable only when Kyvos compute and raw data querying is enabled on the semantic model. While scheduling Process Full Data for a semantic model, the Both option must be selected.

    1. Aggregated Data: The system will keep the aggregated data (and update it in subsequent semantic model processing as per the query patterns) in Kyvos Analytical store. This data will be used to serve the aggregates queries.

    2. Raw Data: The system will keep the raw data in Kyvos Analytical store. This data will be used to serve those queries for which smart aggregates are not available in the Kyvos Analytical store.

    3. Both: The system will keep aggregated and raw data both.

  9. Depending upon the type of Aggregation Strategy selected  in the semantic model design, you can select any of the following:

    1. Generate minimal aggregates: Semantic model will be processed with minimal aggregation strategy (lowest level materialization only). Use this option to compute values for only the lowest levels in the hierarchy.

    2. Generate fresh aggregates: Fresh aggregates definition recommendation will be taken and existing aggregates will be deleted (if any).

    3. Use existing aggregates definition: No new aggregates definition recommendation will be taken, aggregates will be created using the existing definition only.

  10. Add additional query patterns or analyze query patterns using the corresponding links. 

  11. Optionally, click filter queries to select the queries to be used for the aggregation strategy.
    NOTE: This job will overwrite existing semantic model data (if any).

  12. Choose to run once, now, or set up a recurring schedule. With each option, you can also view or change the properties currently in use.

    • If you choose Once, select a date, time, and time zone.

    • To set up a recurring process, enter the start and end times, the frequency, date, days, week, moths, hours, minutes, and time zone.
      Specify whether to job once per day or repeat. Kyvos picks the time zone from where the report server is running, as the default time zone. You can also specify configure semantic model job schedules for specific days of the week using a recurring schedule. This includes flexible options to schedule jobs on non-consecutive days, such as alternating days, specific weekdays, or any custom combination of days.

  13. Click Schedule to start the process.

image-20241125-185824.png
  1. From the Toolbox, choose Semantic Model.

  2. Select a semantic model from the list and click Process

  3. From the top right of the window, click Schedule Job.

  4. From the Job Type list, select the Upgrade Aggregates option.

  5. Select the job connection that you want to use. In case there are multiple job connections in the underlying registered files in the semantic model, semantic model validation will be triggered on the default Hadoop connection.

  6. You can review and alter the design using recommendations.

  7. Select partitions to update aggregates. 

  8. Select Process as Metadata and Data or Metadata only. This is applicable only when Kyvos compute-based processing is enabled on the semantic model.

    • Metadata and Data: Selecting this option will populate both the dimension metadata cache and data (Kyvos Analytical Store). Queries will be served from Kyvos Analytical Store.

    • Metadata Only: The system will populate only dimension metadata cache. Your queries will be submitted directly to the underlying data source.

  9. Click one of the following options to select your materialization strategy.
    NOTE: This is applicable only when Kyvos compute and raw data querying is enabled on the semantic model. While scheduling Process Full Data for a semantic model, the Both option must be selected.

    1. Aggregated Data: The system will keep the aggregated data (and update it in subsequent semantic model processing as per the query patterns) in Kyvos Analytical store. This data will be used to serve the aggregates queries.

    2. Raw Data: The system will keep the raw data in Kyvos Analytical store. This data will be used to serve those queries for which smart aggregates are not available in the Kyvos Analytical store.

    3. Both: The system will keep aggregated and raw data both.

  10. Depending upon the type of Aggregation Strategy selected in the semantic model design, you can select any of the following:

    1. Generate additional aggregates, keeping the existing ones (if any): Additional aggregates definition recommendation will be taken keeping existing aggregates as it is.

    2. Use existing aggregates definition: No new aggregates definition recommendation will be taken, aggregates will be created using the existing definition only.

  11. Add additional query patterns or analyze query patterns using the corresponding links. 

  12. Optionally, click filter queries to select the queries to be used for the aggregation strategy.
    NOTE: This jobwill overwrite existing semantic model data (if any).

  13. Choose to run once, now, or set up a recurring job schedule. With each option, you can also view or change the properties currently in use.

    • Now: This will immediately start the job.

    • Once: This will run the job at specified time. If you choose Once, select a date, time, and time zone.

    • Recurring: This will run the job regularly at the specified time. To set up a recurring process, enter the start and end times, the frequency, date, hours, minutes, and time zone.
      Specify whether to job once per day or repeat. Kyvos picks the time zone from where the report server is running, as the default time zone. You can also specify configure semantic model job schedules for specific days of the week using a recurring schedule. This includes flexible options to schedule jobs on non-consecutive days, such as alternating days, specific weekdays, or any custom combination of days.

  14. Click Schedule to start the process.

image-20241125-185917.png

Other job types

To schedule other job types, click the ... menu on the Process tab and choose Add other jobs option.

Frequency

You can specify the frequency of when your semantic model gets updated using the following options:

For example:

When you choose Full or Full Test data process, you can specify to process the source databases if modified (if changes have occurred) or process always. Depending on the job type you select, you can also define whether to process once per day or repeat. You can also view the properties currently in use or change them.

Resuming after process failures

If you resume a failed process, the steps that were successfully completed are skipped reducing the process time. There are several ways to resume a process after it fails to complete. 

Modifying semantic model design after process

Typically, to modify the semantic model design after a process, you need to perform a reprocess (incremental or full prcoess) for the changes to come into effect. However, you can modify certain entities and the partition and aggregation strategy of a semantic model without the need for reprocessing. See Semantic model modifications allowed without reprocess to know more.

For this, perform the following steps.

  1. Navigate to the semantic model that you want to modify.

  2. Make the necessary changes on the designer screen and click Save.

Rebalancing Aggregates

When aggregates are unbalanced among existing Query Engines, and this impacts query performance. However, from Kyvos 2024.1 onwards, you can rebalance aggregates from the Kyvos Web Portal.

You can rebalance aggregates on Query Engines for all semantic models by using the Re-balance Aggregates option displayed in the Actions menu on the Process tab.

The confirmation message is displayed at the bottom of the Process page of Semantic Model.


Related topics