Applies to: Kyvos Enterprise  Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

As a part of the constant endeavor to enhance the ease of working with Kyvos, the Continuous Integration and Continuous Deployment (CI/CD) Utility is created to automate the end-to-end migration of Kyvos entities (Semantic model, Relationships, Datasets, and Workbooks) from Kyvos Source environment to Kyvos Target environment using Git as a versioning tool.

The CI/CD Utility comprises the following two utilities:

1. The Export Utility is created for the semantic model designers to export the entities specified in the configuration file from Kyvos and push them to the feature branch of the user repository. The Utility is based on a Python script to execute the REST API to export the required entities from Kyvos in the form of XML files, stage into a local folder, and finally, push these changes to a user-specified Git repository.

2. The Deploy Utility is created for the Admin users to fetch entities planned for deployment from the Git repository and deploy the entity XML files to the target Kyvos environment. Jenkins is used for automating the deployment from GIT Repository to Kyvos. Jenkins job triggers the Deploy executable file, which triggers REST API using Python script to Deploy the entity XML files to Kyvos target environment.

Download the Kyvos CICD Utility.

Export Utility

Prerequisites

To use the Export utility, you must have the following.

1. Git Bash: It is a package that installs Bash, some common bash utilities, and Git on a Windows operating system. This is needed to execute the Git commands on your system.

2. Git: Account and Repository with main and feature branch.

Folder structure

Unzip the CICD_Utility folder to browse the Export folder. The following figure shows the folder structure for the Export utility.

Open

The Export folder consists of the following sub-folders.

• Config: This folder contains the json file with all the configuration details of the entities that need to be exported from Kyvos and GIT Repo details.
  Users will need to update this config file to execute the Utility with list of entities that need to be exported.

• GitRepo: This is a staging location to store the entities exported from Kyvos and Push to Git specified repository.

• Output: This folder contains the utility execution logs with a timestamp. It also includes the exported data if the data is exported to local only and no push is needed to Git repository.

• Export_Executor: It contains the executable file for the Export utility, which triggers the end-to-end export process.

Configuration file

Navigate to the Config folder and open the config.json file. The following figure shows a snippet of the file.

Open

The following table provides details of the parameters available in the config.json file.

Below are several scenarios to export multiple semantic models, relationships, and dataset entities in single export using the config.json if the respective values are set as Y.

• #Appendix

  Export single entity present from a single folder

  • "CUBEDETAILS": { "Folder1" : [ “Cube1”] }

  • "DRDDETAILS": { "Folder1" : [ “DRD1”] }

  • "RFDETAILS": { "Folder1" : [ “RF1”] }

  • "WBDETAILS": { "Folder1" : [ “WB1”] }

• Export multiple entities from a single folder

  • "CUBEDETAILS": { "Folder1" : [ “Cube1”, “Cube2”] }

  • "DRDDETAILS ": { "Folder1" : [ “DRD1”, “DRD2”] }

  • "RFDETAILS": { "Folder1" : [ “RF1”, “RF2”] }

  • "WBDETAILS": { "Folder1" : [ “WB1”, “WB2”] }

• Export multiple entities from multiple folders

  • "CUBEDETAILS": { "Folder1" : [ “Cube1”,”Cube2”], “Folder2” : [“Cube3” ] }

  • "DRDDETAILS ": { "Folder1" : [ “DRD1”,” DRD2”], “Folder2” : [“DRD3” ] }

  • "RFDETAILS": { "Folder1" : [ “RF1”,” RF2”], “Folder2” : [“RF3” ] }

  • "WBDETAILS": { "Folder1" : [ “WB1”,” WB2”], “Folder2” : [“WB3” ] }

• Export All entities from one or more folders

  • "CUBEDETAILS": { "Folder1" : [], “Folder2” : [] }

  • "DRDDETAILS ": { "Folder1" : [], “Folder2” : [] }

  • “RFDETAILS": { "Folder1" : [], “Folder2” : [] }

  • “WBDETAILS": { "Folder1" : [], “Folder2” : [] }

• Export single/multiple entities from ROOT (when entities are not a part of any folder)

  • "CUBEDETAILS": { "ROOT" : [ “Cube4”, “Cube5”] }

  • "DRDDETAILS ": { "ROOT" : [ “DRD4”, “DRD5”] }

  • "RFDETAILS": { "ROOT" : [ “RF4”, “RF5”] }

  • "WBDETAILS": { "ROOT" : [ “WB4”, “WB5”] }

• Export ALL entities, which includes all entities at ROOT and all folders with all entities

  • “CUBEDETAILS": { }

  • “DRDDETAILS ": { }

  • “RFDETAILS": { }

  • “WBDETAILS": { }

Executing Export Utility

Once you have provided the configuration file parameters information, you can execute the Export Utility to start exporting objects.

For this, perform the following steps.

1. Browse to the Export_Executor folder and double-click the exe file to execute it.
   
   

2. The Export utility starts executing. Once the export is complete, you can check the execution logs in the output folder.
   The following figure shows a snippet of the Export Utility execution.

3. If the GITPUSH parameter is set as Y in the Config file, then all the entity XML files will get exported to the Git Feature branch in the following order.

   1. A GIT PULL is performed from the feature branch specified in the config file. So that the remote and local repository (GITRepo) are in sync.

   2. The entities are exported to the GITRepo folder from Kyvos, which is a local repository clone of the Git repository, as shown in the following figure.

   3. Entities are then pushed to the Git feature branch, as shown in the following figure. We have used the working feature branch in this case.
      
      

4. If the GITPUSH parameter is set as N in the Config file, then the entities are directly exported to the Output folder in the Export No GIT PULL and GIT PUSH will be performed in this case.

Deploy Utility

Prerequisites

To use the Deploy utility, you must have the following.

1. Git Bash: It is a package that installs Bash, some common bash utilities, and Git on a Windows operating system. This is needed to execute the Git commands on your system.

2. Git: Account and Repository with the main branch.

3. Jenkins: For automated execution of the Deploy utility, based on a trigger from Git.

4. You will need Python 3.x if you want to execute the Python script instead of the executable Utility. The script uses requests, os, sys, datetime, logging, and urllib3 Python packages. See the section Deploying entities using Python script for details on how to execute the Python script.

Folder structure

Unzip the CICD_Utility folder to browse the Deploy folder. The following figure shows the folder structure for the Deploy utility.

The Deploy folder consists of the following sub-folders.

• Output: This folder contains the utility execution logs with a timestamp.

• Deploy_Executor: It contains the executable file for the Deploy utility, which triggers the end-to-end deployment process.

Executing Deploy Utility

Jenkins job Setup, Integration with Git Repository, and Creation of Git Connection in Jenkins

1. In case the Jenkins job is not created and configured, see the section Creating new Project in Jenkins.

2. In case the Jenkins job is not Integrated with Git Repository, see the section Git settings to integrate with Jenkins.

3. In case the Git connection is not created in Jenkins, see the section Creating Git Connection in Jenkins.

Once the above steps are completed, you can execute the Deploy Utility using any of the following methods.

Deploying entities using the executable file

Using this method, you can deploy entities by executing the Deploy Utility through DeployKyvosEntities.exe file.

The file is available at the CICD_Utility\Deploy\Deploy_Executor location, as shown in the following figure.

Trigger the Deploy Utility by executing the following commands using the Jenkins jobs.

cd <Deploy Exe Path> <Deploy Exe File Name> <Jenkins Project workspace> <Kyvos URL> <Kyvos UserName> <Kyvos Password>

Example:

In the command above,

• Deploy Exe Path is the path where the Deploy executable file is available.

• Deploy Exe File Name is the name of the executable file, i.e., exe.

• Jenkins Project workspace is the path of Jenkins workspace for your project. Data from the Git main repository is pulled here using Jenkins. This location acts as a source for deploying the data to the target environment. You will need to execute the Jenkins jobs the first time to create the Jenkins workspace for your new project.

• Kyvos URL is the URL of the target Kyvos environment.

• Kyvos Username is the name of the native user with admin access to the target Kyvos environment.

• Kyvos Password is the password of the native user with admin access to the target Kyvos environment.

The following figure shows the execution of the Deploy Utility from Jenkins job.

The following figure shows Kyvos Monitor showing updated datasets, Relationships, and semantic model entities after the execution of Deploy Utility.

Deploying entities using Python script

You can also deploy entities using the DeployKyvosEntities.py Python script. In this case, the Deploy Utility is triggered by executing the following commands using the Jenkins jobs.

The file is available at the CICD_Utility\Deploy\Deploy_Executor location, as shown in the following figure.

Trigger the Deploy Utility by executing the following commands using the Jenkins jobs.

cd <Deploy Script Path>  python <Deploy Python File Name> <Jenkins Project workspace> <Kyvos URL> <Kyvos Username> <Kyvos Password>

Example:

In the command above,

• Deploy Script Path is the path where the Deploy Python script file is available.

• Deploy Python File Name is the name of the Python script, i.e., py.

• Jenkins Project workspace is the path of Jenkins workspace for your project. Data from the Git main repository is pulled here using Jenkins. This location acts as a source for deploying the data to the target environment. You will need to execute the Jenkins jobs the first time to create the Jenkins workspace for your new project.

• Kyvos URL is the URL of the target Kyvos environment.

• Kyvos Username is the name of the native user with admin access to the target Kyvos environment.

• Kyvos Password is the password of the native user with admin access to the target Kyvos environment.

The following figure shows the execution of the Deploy Utility from Jenkins job using Python script.

Appendix

Creating Personal Access Token in Git

The Personal Access Token is required for authentication of Export Utility with Git. You are required to provide this in the Export Utility config file, as explained in the Export Utility section.

To create a Personal Access Token in Git, perform the following steps.

1. In your Git account, navigate to Edit Profile.

2. On the left pane, click Access Tokens.

3. Give the token a Descriptive name, select the Scopes which define the access for the Personal tokens, and then click the Create personal access token. A token is generated. 

Creating Git Connection in Jenkins

To connect Jenkins with Git, create a connection by following steps:

1. In Manage Jenkins > System Configuration > Configure System and then follow these steps.

   1. In the Git section, check the Enable authentication for '/project' end-point

   2. Click Add, then choose Jenkins Credential Provider.

   3. Choose Git API token as the token type.

   4. Enter the Git personal access token's value in the API Token field and click Add.

   5. Enter the Git server's URL in the Git host URL.

   6. Click Test Connection, ensuring the connection is successful before proceeding.

   7. After testing the connection, enter the connection name and save.

Creating new project in Jenkins

To create a new project in Jenkins, perform the following steps.

1. Login to your Jenkins account and click New Item from the top left of your dashboard.

2. Enter the name of the item you want to create, select Freestyle project, and click OK. The project is created.
   
   

3. Navigate to the newly created project and select the Configure option, as shown in the following figure.
   
   

    

4. On the General tab, click on Git Connection and select the connection.

5. On the Source Code Management tab, select the Git option on the left and provide your Git Repository URL and Credentials in the corresponding fields.

6. Change the Branch Specifier as shown in the below screenshot */main this is the branch we intend to process using Jenkins. In this example, we have selected main to Process.

7. Select the Repository Browser as Auto.

8. In Additional Behaviours, select the Wipe out repository & force clone option to erase the local repository and clone the fresh repository every time to avoid inconsistency.

9. On the Build Triggers tab, select Build when a change is pushed to Git

10. It is used to execute the Deploy Utility as soon entities are merged from the Feature branch to the Master branch in Git repository mentioned in Jenkins above.
    
    

Creating secret token in Jenkins

1. Click Advanced to generate the secret token required for configuring Webhooks.

2. On the Build Steps tab, select the Execute Window batch command and enter the command (explained in the Deploy Utility section) to execute the Utility.

3. Click the Save button to save your changes.

Git settings to integrate with Jenkins

Webhooks trigger the Jenkins job to execute the Deploy Utility if the defined event happens.

To integrate Git with Jenkins, perform the following steps.

1. In your Git Account, select the repository you want to integrate with Jenkins and go to Settings.

2. Select the Webhooks option from the left pane.

3. In the URL, enter the URL and append “/project/project name” to this URL, as shown in the figure below, and enter the secret token. To generate the secret token, check the section Creating secret token in Jenkins). For example, http://localhost:8080/project/Git

4. Select Push events in the trigger and then click Add webhook.