Introduction

As a part of the constant endeavor to enhance the ease of working with Kyvos, the Continuous Integration and Continuous Deployment (CI/CD) Utility is created to automate the end-to-end migration versioning of Kyvos entities (like Semantic Models, DatasetRelationships, and RegisterDataset) from Kyvos Source environment to Kyvos Target environment Relationships, Datasets, etc., using Git as a versioning tool.

The CI/CD Utility comprises of the following two utilities:

• The Export Utility is created for the semantic model designers to export the entities specified in the configuration file from Kyvos and push them to the feature branch of the user repository. The Utility is based on a Java to execute the REST API to export the required entities from Kyvos in the form of XML files, stage into a local folder, and finally, push these changes to a user-specified Git repository.

• The Deploy Utility is created for the Admin users to fetch entities planned for deployment from the Git repository and deploy the entity XML files to the target Kyvos environment. Jenkins is used for automating the deployment from GIT Repository to Kyvos. Jenkins job triggers the Deploy executable file, which triggers REST API using java to Deploy the entity XML files to Kyvos target environment.

Prerequisites

To use the Export utility, you must have the following.

Git Bash: It is a package that installs Bash, some common bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system.

GitLab: Account and Repository with main and feature branch.

Setup details 

The utility setup is provided as a single .zip file (Linux) that you can unzip at any location on any computer.

On extracting the setup, you will see the following folders and files: 

Linux computer: 

1.conf 

2.input 

3.lib 

4.ExportKyvosEntities.sh 

5.DeployKyvosEntities.sh

6.output

Setting up parameters for CICD Utility

1. In the Linux machine and navigate to where the setup files are extracted. 

2. Navigate to the conf folder and open the serverDetail.properties file for editing. Here, provide the values for the following parameters, and save the file. 

   • KYVOS_URL= http://<Server IP>:8081/kyvos/rest/ 

   • USERNAME= <Name of user> 

   • PASSWORD= <Kyvos URL password> 

3. Navigate to the input folder and open the input.json file for editing. Here, provide the values for the following parameters, and save the file. 

Snippet of input.json file

It can be used to migrate entities from one Kyvos deployment (Source) to another Kyvos deployment (Target). It uses Gitlab to store versions of Kyvos entities in XML format.  

• It also provides integration with Bitbucket repositories.  

• It is a Java-based utility and uses Kyvos REST APIs to export the entities from source Kyvos deployment and to import the entities to target Kyvos deployment. It is designed to be used in Linux system. 

• It is recommended to use Kyvos user credentials having administrative privileges to import and export all sorts of entities. 

The CI/CD Utility comprises the following two utilities:

• Export Utility: It fetches entities from Kyvos deployment in XML format, stores them in a local folder, and then pushes them to the configured Git repository. It has a configuration file to define all entities that are to be fetched along with source Kyvos deployment and git repository details.

• Import Utility: It uses Jenkins to automate the import of entities from Git repository to target Kyvos deployment.  Jenkins pipeline is configured in Gitlab server to pull the entities from Gitlab to the local file system. Jenkins then launches the import utility shell script which then finally imports the entities from file system to target Kyvos deployment.

Utility bundle details  

The utility bundle contains a single zip file having following folders and files within it:  

• Conf folder: It contains serverDetailExportUtility.properties , serverDetailImportUtility.properties and cicdLogs.properties 

• Input folder contains input.json file to be used for configuring the entities to be exported and Gitlab server details. 

• Lib folder: folder containing required libraries for the utility 

• output folder: initially blank folder and to be used for placing exported entities XML file when not pushing into Gitlab server. 

• ExportKyvosEntities.sh: Script file to launch Export utility 

• ImportKyvosEntities.sh: Script file to launch Import utility 

Export Utility 

Prerequisites 

1. Git Bash: It is a package that installs Bash, some common bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system. 

2. GitLab: Git server with main and feature branches. 

Setting up Export Utility 

1. Extract the setup zip.  

2. Configure source Kyvos deployment details in serverDetailExportUtility.properties file in Conf folder. This file needs the details below to be filled in. If the details are not provided in this file, then alternatively these details can be provided as an argument to the export utility script. 

• KYVOS_URL = <YourKyvosURL>/rest/ 

• USERNAME = <KyvosUserName> 

• PASSWORD = <KyvosUserPassword> 

3. Configure input.json within input folder to define the entities to be exported.  The following table provides details of the parameters

...

3. in the input.json file.

...

Parameter

...

Description

SEMANTIC_MODEL_DETAILS

3.  

4. Provide

...

Scenarios and examples are mentioned after this table in cases

where multiple entities need to be exported.

...

DATASET_RELATIONSHIP_DETAILS

...

3. the execute permission on the ExportKyvosEntities.sh file, if not already provided (chmod +x ExportKyvosEntities.sh) 

...

Below are several scenarios to export multiple semantic models, datasetrelationships, and dataset entities in single export using the input.json if the respective values are set as Y.

...

Export single entity present from a single folder

• "SEMANTIC_MODEL_DETAILS": { "Folder1" : [ “Model1”] }

• "DATASET_RELATIONSHIP_DETAILS": { "Folder1" : [ “DatasetRelationship1”] }

• "DATASET_DETAILS": { "Folder1" : [ “Dataset1”] }

Export multiple entities from a single folder

...

...

Example configurations for different scenarios

• Export All entities from one or more folders

"SEMANTIC_MODEL_DETAILS": {"Folder1" : [

...

],

...

“Folder2”: [] }

"DATASET_RELATIONSHIP_DETAILS": { "Folder1" : [

...

],

...

“Folder2” : [] }

...

“DATASET_DETAILS": { "Folder1" : [

...

],

...

“Folder2” : [] }

• Export single/multiple entities from multiple foldersROOT (when entities are not a part of any folder)

"SEMANTIC_MODEL_DETAILS": { "

...

ROOT" : [

...

“Model4”, “Model5”] }

"DATASET_RELATIONSHIP_DETAILS": {"

...

ROOT": [

...

“DatasetRelationship4”,

...

“DatasetRelationship5”]}

"DATASET_DETAILS": { "

...

ROOT" : [

...

“Dataset4”, “Dataset5”] }

• Export All entities from one or more folders

  "SEMANTIC

  ALL entities, which includes all entities at ROOT and all folders with all entities

“SEMANTIC_MODEL_DETAILS": {

...

}

...

“ DATASET_RELATIONSHIP_DETAILS

...

": {

...

}

“DATASET_DETAILS": {

...

}

...

Export single/multiple entities from ROOT (when entities are not a part of any folder)

• "SEMANTIC_MODEL_DETAILS": { "ROOT" : [ “Model4”, “Model5”] }

• "DATASET_RELATIONSHIP_DETAILS": { "ROOT" : [ “DatasetRelationship4”, “DatasetRelationship5”] }

• "DATASET_DETAILS": { "ROOT" : [ “Dataset4”, “Dataset5”] }

...

Export ALL entities, which includes all entities at ROOT and all folders with all entities

• “SEMANTIC_MODEL_DETAILS": { }

• “ DATASET_RELATIONSHIP_DETAILS ": { }

• “DATASET_DETAILS": { }

Executing ExportKyvosEntities on Linux 

To execute the utility on a Linux computer, perform the following steps. 

1. In the Linux machine, go to the folder where the utility was unzipped. 

2. Provide the execute permission on the ExportKyvosEntities.sh file, if not already provided (chmod +x ExportKyvosEntities.sh)

3. Run the ./ExportKyvosEntities.sh command. 

You will see logging, semantic model validation, and success messages on the successful execution of the utility. In case of any exception, you will see failed messages, and all newly created folders will be deleted. Please get in touch with the Kyvos support team in case of an exception. 

ExportKyvosEntities.shfile

1. The Export utility starts executing. Once the export is complete, you can check the execution logs in the output folder.
   The following figure shows a snippet of the Export Utility execution.

   Image Removed

2. If the GITPUSH parameter is set as Y in the input file, then all the entity XML files will get exported to the GitLab Feature branch in the following order.

   1. A GIT PULL is performed from the feature branch specified in the input file. So that the remote and local repository (GITRepo) are in sync.

   2. The entities are exported to the GITRepo folder from Kyvos, which is a local repository clone of the GitLab repository.

   3. Entities are then pushed to the GitLab feature branch; we have used the working feature branch in this case

3. If the GITPUSH parameter is set as N in the input file, then the entities are directly exported to the Output folder. No GIT PULL and GIT PUSH will be performed in this case.

Deploy Entities Utility 

We can deploy entities using two methods

1.  We can deploy through shell file when entities present in local machine

2.  We can deploy using jenkins pipeline which configure with gitlab

Prerequisites

To use the Deploy utility, you must have the following.

• Git Bash: It is a package that installs Bash, some common bash utilities, and Git on  operating system. This is needed to execute the Git commands on your system.

• gitlab: Account and Repository with the main branch.

• Jenkins: For automated execution of the Deploy utility, based on a trigger from gitlab.

 Deploying entities using shell file

Parameters for Deploy Utility

We need to pass these parameters while running deploy utility on Linux  on command line

KYVOS_URL= http://<Server IP>:8081/kyvos/rest/ 

USERNAME= <Kyvos Username> 

PASSWORD= <Kyvos User Password> 

KyvosEntitiesPath=<Path of KyvosEntities store in local machine>

Executing deploy on Linux 

To execute the utility on a Linux computer, perform the following steps. 

1. In the Linux machine Go to the folder where the utility was unzipped. 

2. Provide the execute permission on the DeployKyvosEntities.sh file, if not already provided (chmod +x DeployKyvosEntities.sh). 

3. Run the ./DeployKyvosEntities.sh $KyvosUrl $Username $Password $KyvosEntitiesPath command. 

You will see logging, semantic model validation, and success messages on the successful execution of the utility. In case of any exception, you will see failed messages, and all newly created folders will be deleted. Please get in touch with the Kyvos support team in case of an exception. 

Executing Deploy Utility from jenkins

Creating new Project in Jenkins. In case the Jenkins job is not Integrated with GitLab Repository, see the section Gitlab settings to integrate with Jenkins. Once the above steps are completed, you can execute the Deploy Utility.

Trigger the Deploy Utility by executing the following pipeline using the Jenkins jobs.

sh "java -cp <CICD utility jar file path>:<Utility commons jar file path> com.kyvos.deploy.DeployKyvosEntities ${sourceKyvosURL} ${sourceKyvosUser} ${sourceKyvosPassword} ${kyvosEntitiesPath}"

Create a pipeline mention in below image

...

In the command above,

Add these parameters in Jenkins pipeline:

• Click on project as parameter then add parameter given below

• kyvosEntitiesPath represents the directory path storing Kyvos entities within the Jenkins workspace dedicated to your project. During the Jenkins workflow, data is fetched from the main GitHub repository and stored in this location. Subsequently, this directory serves as the source for deploying data to the intended environment. It's crucial to run the Jenkins jobs initially to establish the workspace for your new project.

• KyvosURL is the URL of the target Kyvos environment.

• KyvosUsername is the name of the native user with admin access to the target Kyvos environment.

• KyvosPassword is the password of the native user with admin access to the target Kyvos environment.

The job is automatically triggered by webhook whenever you push entities to main branch.

The following figure shows Kyvos Monitor showing updated Files, Relationship, and Cube Entity after the execution of Deploy Utility.

...

Audit logs of entities updated in Kyvos

Appendix

The Personal Access Token is required for authentication of Export Utility with Git. You are required to provide this in the Export Utility config file, as explained in the Export Utility section.

To create a Personal Access Token in GitLab, perform the following steps.

1. In your GitLab account, navigate to Settings.

2. On the left pane, click the Developer settings option, select the Personal access tokens from the left pane, and click Generate new token.

   Image Removed

3. Give the token a Descriptive name, select the Scopes which defines the access for the Personal tokens, and then click the Generate token.

4. A token is generated. Make sure you copy your new personal access token to be used in Export Utility.

   Image Removed

5. You can also Update, Regenerate, Revoke, or Delete the token using the options available on the page.

Creating new Project in Jenkins

To create a new project in Jenkins, perform the following steps.

1. Login to your Jenkins account.

2. Install the following plugins:

   1. Git client plugin

   2. GitHub API

   3. GitHub Branch Source Plugin

   4. GitHub plugin

   5. Genric Webhook plugin

3. If the plugins are not installed while initializing Jenkins. Go to Manage Jenkins >Manage Plugins and install the plugins.

4. Click New Item from the top left of your dashboard.

5. Enter the name of the item you want to create, select Pipeline , and click OK. The project is created.

   Image Removed

6. Navigate to the newly created project and select the Configure option, as shown in the following figure.

   Image Removed

7. On the General tab, select the GitHub Project check box and provide the URL of the GitHub/GitLab repository in the Project url field, as shown in the following figure.

   Image Removed

8. On the Source Code Management tab, select the Git option on the left and provide your Git Repository URL and Credentials in the corresponding fields.

9. Change the Branch Specifier as shown in the below screenshot refs/head/{nameofbranch} this is the branch we intend to build using Jenkins. In this example, we have selected main to Build.

10. Select the Repository Browser as Auto.

11. In Additional Behaviours, select the Wipe out repository & force clone option to erase the local repository and clone the fresh repository every time to avoid inconsistency.

12. On the Build Triggers tab, select GitHub hook trigger for GITScm polling option. This triggers a Jenkins job when the GitHub webhook relays any payload or If you are using genric webhook use can select on genric webhook trigger and genrate a token .

13. It is used to execute the Deploy Utility as soon entities are merged from the Feature branch to the Master branch in GitHub repository mentioned in Jenkins above.
    

    Image Removed

15.   On the Build tab, select the Pipeline command and create the pipeline (explained in the Deploy Utility section) to execute the Utility.

...

Executing Export Utility

Run ./ExportKyvosEntities.sh $KyvosUrl, $Username, $Password 

The parameters are optional, if set in serverDetailExportUtility.properties file. If values are given at both places, script arguments take the preference.

You can access the execution log file located within the "conf/log" folder, which contains the file named "cicd.log".

Steps Performed by Export Utility

1. If the GITPUSH parameter is set as Y, then all the entity XML files will get exported to the GitLab Feature branch in the following order.

   1. A GIT PULL is performed from the feature branch specified in the input file. So that the remote and local repository (GITRepo) are in sync.

   2. The entities are exported to the GITRepo folder from Kyvos, which is a local repository clone of the GitLab repository.

   3. Entities are then pushed to the GitLab feature branch.

2. If the GITPUSH parameter is set as N in the input file, then the entities are directly exported to the Output folder. No GIT PULL and GIT PUSH will be performed in this case.

Import Utility

The import utility works in two steps.

Step 1: On changes in configured Gitlab branch, Jenkins pipeline triggers to pull the changed files in local file system, it then launches the ImportKyvosEntities.sh script.

Jenkins pipeline is required to automate

1. the changes identification in configured Gitlab branch and

2. pull the changed entity files in configured folder within local file system

3. to launch the ImportKyvosEntities.sh script

Step 2: On execution ImportKyvosEntities.sh script reads all the entities xml files placed in configured folder and uses Kyvos REST APIs to import them in target Kyvos deployment.

In case Jenkins pipeline is not configured, then Step 2 can still work on launching ImportKyvosEntities.sh independently.

Prerequisites

• Git Bash: It is a package that installs Bash, some common bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system.

• GitLab: Git server with main and feature branches.

• Jenkins: For automated execution of the Import utility script, based on a trigger from Gitlab.

Setting up Import Utility

1. Extract the setup zip. 

2. Configure source target Kyvos deployment details in serverDetailImportUtility.properties file in Conf folder. This file needs the details below to be filled in. If the details are not provided in this file, then alternatively these details can be provided as an argument to the import utility script.

   • KYVOS_URL = <YourKyvosURL>/rest/

   • USERNAME = <KyvosUserName>

   • PASSWORD = <KyvosUserPassword>

3. Provide the execute permission on the ImportKyvosEntities.sh file, if not already provided (chmod +x ImportKyvosEntities.sh)

Executing Import Utility separately without Jenkins to perform step 2

Run ./ImportKyvosEntities.sh $KyvosEntitiesPath $Username $Password $KyvosUrl 

KyvosEntitiesPath is a local machine’s GitLab Repo folder path where entities are stored on pull. This is mandatory parameter. The other parameters are optional, if set in serverDetailImportUtility.properties file. If values are given at both places, script arguments take the preference.

You can access the execution log file located within the "conf/log" folder, which contains the file named "cicd.log".

Jenkins job setup and integration with GitLab Repository

In case the Jenkins job is not created and configured, see the section Creating new Project in Jenkins.

In case the Jenkins job is not Integrated with GitLab Repository, see the section Gitlab settings to integrate with Jenkins.

Once the above steps are completed, you can execute the Import Utility via Jenkins.

Executing Import Utility via Jenkins

Trigger the Import utility by executing the following pipeline using the Jenkins job.  Once configured, the job will automatically be triggered by webhook whenever you push entities to main branch.

sh "jenkinsWorkspace/ImportUtilityFilePath ${KyvosEntitiesPath} ${KyvosUsername} ${KyvosPassword} ${KyvosURL}" 

All the parameters are mandatory here. Add these parameters in Jenkins pipeline:

• KyvosEntitiesPath: represents the directory path storing Kyvos entities within the Jenkins workspace dedicated to your project. During the Jenkins workflow, data is fetched from the main GitLab repository and stored in this location. Subsequently, this directory serves as the source for deploying data to the intended environment. It's crucial to run the Jenkins jobs initially to establish the workspace for your new project

• KyvosUsername is the name of the native user with admin access to the target Kyvos deployment.

• KyvosPassword is the password of the native user with admin access to the target Kyvos deployment.

• KyvosURL is the URL of the target Kyvos deployment.

Jenkins pipeline

...

pipeline {
    agent any
    stages {
        stage ('Environment Details') {
            steps {
                sh '''
                env | grep -e PATH -e JAVA_HOME
                which java
                java -version
                mvn –version"” }}
             stage('Checkout') {
            steps {
                checkout (
                    [$class: 'GitSCM',
                    branches: [[name: '*/your_branch_name']],
                    extensions: [], userRemoteConfigs:
                    [[credentialsId: 'your_credential_id',
                    url: 'your_git_url']]]   ) } }  
        stage('Build') {
            steps {
                script{
                 
                    dir('jenkinsWorkSpacePath/cicdUtilityPath') {
                        sh "pwd"
                sh " jenkinsWorkSpacePath /cicdUtilityPath/ImportKyvosEntities.sh ${kyvosEntitiesPath} ${KyvosUsername} ${kyvosPassword} ${KyvosURL}"     }  } } } } }

• Kyvos Monitor Image shows updated audit entries for imported Datasets, Relationship, and Semantic Model entities after the execution of Import utility.

  Image Added

Appendix

Creating Personal Access Token in GitLab

The Personal Access Token is required for authentication of Export Utility with Git. You are required to provide this in the Export Utility config file, as explained in the Export Utility  section. To create a Personal Access Token in GitLab, perform the following steps.

To create a Personal Access Token in GitLab, perform the following steps.

1. In your GitLab account, navigate to Settings.

2. On the left pane, click the Developer settings option, select the Personal access tokens from the left pane, and click Generate new token.

   Image Added

3. Give the token a Descriptive name, select the Scopes which defines the access for the Personal tokens, and then click the Generate token.

4. A token is generated. Make sure you copy your new personal access token to be used in Export Utility.

   Image Added

5. You can also Update, Regenerate, Revoke, or Delete the token using the options available on the page.

Creating new Project in Jenkins

To create a new project in Jenkins, perform the following steps.

1. Login to your Jenkins account.

2. Install the following plugins:

   1. Git client plugin

   2. GitHub API

   3. GitHub Branch Source Plugin

   4. GitHub plugin

   5. Genric Webhook plugin

3. If the plugins are not installed while initializing Jenkins, go to Manage Jenkins >Manage Plugins and install the plugins.

4. Click New Item from the top left of your dashboard.

5. Enter the name of the item you want to create, select Pipeline , and click OK. The project is created.

   Image Added

6. Navigate to the newly created project and select the Configure option, as shown in the following figure.

   Image Added

7. On the General tab, select the GitHub Project check box and provide the URL of the GitHub/GitLab repository in the Project url field, as shown in the following figure.

   Image Added

8. On the Build Triggers tab, select Generic Webhook Trigger. This triggers a Jenkins job when the GitLab webhook relays any payload. If you are using a generic webhook you can select the generic webhook trigger option.

   Image Added

9. Provide Jenkins credential id in the Token section or generate a token id in Token Credential section by clicking on ADD button and provide credentials.

   Image Added

10. It is used to execute the Import Utility as soon entities are merged from the Feature branch to the Master branch in GitHub repository mentioned in Jenkins above.

11. On the Build tab, select the Pipeline Script command and create the pipeline (explained in the above section) to execute the Utility.

12. Click the Save button to save your changes

GitLab settings to integrate with Jenkins

Webhooks act as a trigger for the Jenkins job to execute the Import Utility if the defined event happens.

To integrate GitLab with Jenkins, perform the following steps.

1. In your GitLab Account, select the repository that you want to integrate with Jenkins and go to the Settings tab.

2. Select the Webhooks option from the left pane.
   

   Image Added

3. Click the Add new webhook option.

4. In the URL field, enter the Jenkins URL. You must add /generic-webhook-trigger/invoke to this and provide secret token credential id or that you have generated in jenkins while configuring webhooks, as shown in the figure below.
   

   Image Added

5. Select the Pushes in the events and click the Add Webhook option.
   

   Image Added

6. Select event Push wildcard expression in that mention the main branch name

7. Click the Save changes option to apply the changes.

8. Then, click on the Test option to test the webhook.