Kyvos CICD Utility

As a part of the constant endeavor to enhance the ease of working with Kyvos, the Continuous Integration and Continuous Deployment (CI/CD) Utility is created to automate the versioning of Kyvos entities like Semantic Models, Relationships, Datasets, etc., using Git as a versioning tool. It can be used to migrate entities from one Kyvos deployment (Source) to another Kyvos deployment (Target).

It uses Gitlab to store versions of Kyvos entities in XML format.
It also provides integration with Bitbucket repositories.
It is a Java-based utility that uses Kyvos REST APIs to export the entities from source Kyvos deployment and import them to target Kyvos deployment. It is designed to be used in Linux systems.

Note

It is recommended to use Kyvos user credentials with administrative privileges to import and export all entities.

The CI/CD Utility comprises the following two utilities:

Export Utility: It fetches entities from Kyvos deployment in XML format, stores them in a local folder, and then pushes them to the configured Git repository. It has a configuration file to define all entities to be fetched along with source Kyvos deployment and git repository details.
Import Utility: It uses Jenkins to automate the import of entities from the Git repository to target Kyvos deployment. The Jenkins pipeline is configured in the Gitlab server to pull the entities from Gitlab to the local file system. Jenkins then launches the import utility shell script, which finally imports the entities from the file system to target Kyvos deployment.

Utility bundle details 

The utility bundle contains a single zip file having the following folders and files within it: 

Conf folder: It contains serverDetailExportUtility.properties, serverDetailImportUtility.properties, and cicdLogs.properties
Input folder: It contains an input.json file to configure the exported entities and the Gitlab server details.
Lib folder: This folder contains the required libraries for the utility.
Output folder: This initially blank folder will place exported entities' XML files when not pushing them into the Gitlab server.
ExportKyvosEntities.sh: This is a script file to launch the Export utility.
ImportKyvosEntities.sh: This is a script file to launch the Import utility.

Export Utility

Prerequisites

Git Bash: It is a package that installs Bash, some common Bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system.
GitLab: Git server with main and feature branches.

Setting up Export Utility

Extract the setup zip file. 
Configure source Kyvos deployment details in the serverDetailExportUtility.properties file located in the Conf folder. Provide the following details in this file. If you don't provide the details in the file, you can alternatively provide them as an argument to the export utility script.
- KYVOS_URL = <YourKyvosURL>/rest/
- USERNAME = <KyvosUserName>
- PASSWORD = <KyvosUserPassword>
Configure input.json within the input folder to define the entities to be exported.  The following table provides details of the parameters in the input.json file.
Provide execute permission on ExportKyvosEntities.sh file, using chmod +x ExportKyvosEntities.sh, if not already provided.

Parameter	Description
EXPORT_SEMANTIC_MODELS	Defines whether to export semantic model entities or not. Y to allow N to deny/refrain
SEMANTIC_MODEL_DETAILS	Provides details of the semantic model that needs to be exported in the following format: "FolderName":["SemanticModelName"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "SemanticModelName": Comma separated entity names existing within the folder Example {"Folder1": [ “Model1”,” Model2”], “Folder2”: [“Model3”]}
EXPORT_DATASET_RELATIONSHIPS	Defines whether to export Dataset Relationship entities or not. Y to allow N to deny/refrain
DATASET_RELATIONSHIP_DETAILS	Provides details of the Dataset Relationship that must be exported in the following format. "FolderName":["DatasetRelationshipName"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "DatasetRelationshipName": Comma separated entity names existing within the folder For example, {"Folder1": [ “DRelationship1”,” DRelationship2”], “Folder2”: [“DRelationship3”]}
EXPORT_DATASETS	Defines whether to export Dataset entities or not. Y to allow N to deny/refrain
DATASET_DETAILS	Provides details of the Dataset that must be exported in the following format. "FolderName": ["DatasetName"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "DatasetName": Comma separated entity names existing within the folder Example {"Folder1": [ “Dataset 1”,” Dataset 2”], “Folder2”: [“Dataset 3”]}
EXPORT_WB	Defines whether to export Workbooks or not. Y to allow N to deny/refrain
WB_DETAILS	Provides details of the Workbooks that must be exported in the following format. "FolderName": ["Workbook"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "Workbook": Comma separated entity names existing within the folder For example, {"Folder1": [ “workbook1”,” workbook2”], “Folder2”: [“workbook3”]}
GIT_PUSH	Defines whether to push the exported entities into GitLab or not. Y to push in GitLab as well as in GIT_LOCAL_REPO_PATH both. N to not push in Gitlab, but just only in the output folder.
GIT_URL	Server URL
GIT_LOCAL_REPO_PATH	Provides a folder path to be used as a staging location. For example, “C:/CICDUtility/Export/GITRepo” NOTE: Use the forward-slash (/) in the repo path.
GIT_FEATURE_BRANCH_NAME	Specifies the GitLab feature branch name where entities will get exported.
GIT_FEATURE_BRANCH_CLEAN	Defines whether to clean the GIT Feature branch before a fresh export. The default value is N. Y to clean N to do nothing
GIT_REPONAME	Specifies the GitLab repository name where the entities will get exported.
GIT_USERNAME	Specifies the GitLab user name to connect with GIT Repo.
GIT_TOKEN	The Personal Access Token for authentication with the user’s GIT account. If you do not have the Personal Access token, you can create it using the steps explained in the Appendix.
GIT_URL	Git url to be mentioned

Note

To export an entity from the root folder, users must specify the folder name as ‘ROOT’ in the input.json file.

Example configurations for different scenarios

Export All entities from one or more folders

"SEMANTIC_MODEL_DETAILS": {"Folder1" : [], “Folder2”: [] }

"DATASET_RELATIONSHIP_DETAILS": { "Folder1" : [], “Folder2” : [] }

“DATASET_DETAILS": { "Folder1" : [], “Folder2” : [] }

Export single/multiple entities from ROOT (when entities are not a part of any folder)

"SEMANTIC_MODEL_DETAILS": { "ROOT" : [ “Model4”, “Model5”] }

"DATASET_RELATIONSHIP_DETAILS": {"ROOT": [ “DatasetRelationship4”, “DatasetRelationship5”]}

"DATASET_DETAILS": { "ROOT" : [ “Dataset4”, “Dataset5”] }

Export ALL entities, which includes all entities at ROOT and all folders with all entities

“SEMANTIC_MODEL_DETAILS": { }

“ DATASET_RELATIONSHIP_DETAILS ": { }

“DATASET_DETAILS": { }

Executing Export Utility

Run ./ExportKyvosEntities.sh $KyvosUrl, $Username, $Password

The parameters are optional and can be set in the serverDetailExportUtility.properties file. If values are given in both places, the script arguments take precedence.

You can access the execution log file in the conf/log folder containing cicd.log.

Steps performed by Export Utility

If the GITPUSH parameter is set as Y, all the entity XML files will be exported to the GitLab Feature branch in the following order.
1. A GIT PULL is performed from the feature branch specified in the input file to sync the remote and local repository (GITRepo).
2. The entities are exported to the GITRepo folder from Kyvos, a local repository clone of the GitLab repository.
3. Entities are then pushed to the GitLab feature branch.
The entities are directly exported to the Output folder if the GITPUSH parameter is set as N in the input file. No GIT PULL and GIT PUSH will be performed in this case.

Note

Upon the initial setup configuration, users are required to execute the following commands:

git config --global user.email "email"
git config --global user.name "username"

Import Utility

The import utility works in two steps.

Step 1: On changes in the configured Gitlab branch, the Jenkins pipeline triggers to pull the changed files in the local file system; it then launches the ImportKyvosEntities.sh script.

A Jenkins pipeline is required to automate

the identification of the change in the configured Gitlab branch and
pull the changed entity files into a configured folder within the local file system
to launch the ImportKyvosEntities.sh script

Step 2: On execution, the ImportKyvosEntities.sh script reads all the entities xml files placed in the configured folder and uses Kyvos REST APIs to import them in target Kyvos deployment.

If the Jenkins pipeline is not configured, Step 2 can still launch ImportKyvosEntities.sh independently.

Prerequisites

Git Bash: It is a package that installs Bash, some common bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system.
GitLab: Git server with main and feature branches.
Jenkins: For automated execution of the Import utility script, based on a trigger from GitLab.

Setting up Import Utility

Extract the setup zip.
Configure source target Kyvos deployment details in the serverDetailImportUtility.properties file in the Conf folder. This file needs the details below to be filled in. If the details are not provided in this file, then alternatively, these details can be provided as an argument to the import utility script.
- KYVOS_URL = <YourKyvosURL>/rest/
- USERNAME = <KyvosUserName>
- PASSWORD = <KyvosUserPassword>
- KyvosEntitiesPath=<Path of KyvosEntities store in local machine>
  NOTE: It’s value should represent the directory path leading up to the KyvosEntities folder.
  For example:
  /data/Automation_Data/mayank/CICD/CICD_Utility_2023.5/EntitiesInMultipleFolders/CICD_Testing/KyvosEntities
Execute the chmod +x ImportKyvosEntities.sh permission on the ImportKyvosEntities.sh file, if not already provided.

Executing Import Utility separately without Jenkins

Execute the following Import Utility separately without Jenkins to perform step 2.

Run ./ImportKyvosEntities.sh $KyvosEntitiesPath $Username $Password $KyvosUrl

KyvosEntitiesPath is a local machine’s GitLab Repo folder path, storing entities on the pull. This is a mandatory parameter. The other parameters are optional if set in the serverDetailImportUtility.properties file. If values are given at both places, script arguments take the preference.

You can access the execution log file in the conf/log folder, which contains the file named cicd.log.

Jenkins job setup and integration with GitLab Repository

If the Jenkins job is not created and configured, see the Creating new Project in Jenkins section.

If the Jenkins job is not Integrated with the GitLab Repository, see the GitLab settings to integrate with Jenkins section.

Once the above steps are completed, you can execute the Import Utility via Jenkins.

Executing Import Utility via Jenkins

Trigger the Import utility by executing the following pipeline using the Jenkins job. Once configured, the job will automatically be triggered by webhook whenever you push entities to the main branch.

sh "jenkinsWorkspace/ImportUtilityFilePath ${KyvosEntitiesPath} ${KyvosUsername} ${KyvosPassword} ${KyvosURL}"

All the parameters are mandatory here. Add these parameters to Jenkins pipeline:

KyvosEntitiesPath: represents the directory path storing Kyvos entities within the Jenkins workspace dedicated to your project. Data is fetched from the main GitLab repository during the Jenkins workflow and stored in this location. Subsequently, this directory is the source for deploying data to the intended environment.

KyvosUsername is the name of the native user with admin access to the target Kyvos deployment.
KyvosPassword is the password of the native user with admin access to the target Kyvos deployment.
KyvosURL is the URL of the target Kyvos deployment.

Kyvos Monitor Image displays updated audit logs for imported Datasets, Relationships, and Semantic Model entities post Import utility execution.

Appendix

Creating Personal Access Token in GitLab

The Personal Access Token is required for authentication of Export Utility with Git. You are required to provide this in the Export Utility config file, as explained in the Export Utility section. To create a Personal Access Token in GitLab, perform the following steps.

To create a Personal Access Token in GitLab, perform the following steps.

In your GitLab account, navigate to Settings.
In the left pane, click the Developer settings >Personal access tokens. The Personal access tokens section is displayed. Click Generate new token.
Specify the token a Descriptive name, select the Scopes which defines the access for the Personal tokens, and then click Generate token. A token is generated.

You can also Update, Regenerate, Revoke, or Delete the token using the options available on the Personal access token page.

Creating new Project in Jenkins

To create a new project in Jenkins, perform the following steps.

Login to your Jenkins account.
Install the following plugins:
1. Git client plugin
2. GitHub API
3. GitHub Branch Source Plugin
4. GitHub plugin
5. Genric Webhook plugin
If the plugins are not installed while initializing Jenkins, go to Manage Jenkins >Manage Plugins and install the plugins.
Click New Item from the top left of your Dashboard.
Enter the name of the item you want to create, select Pipeline, and click Ok. The project is created.
Navigate to the newly created project and select the Configure option, as shown in the following figure.
On the General tab, select the GitHub Project check box and provide the URL of the GitHub/GitLab repository in the Project url field, as shown in the following figure.
On the Build Triggers tab, select Generic Webhook Trigger. This triggers a Jenkins job when the GitLab webhook relays any payload. If you are using a generic webhook, you can select the generic webhook trigger option.
Provide Jenkins credential ID in the Token section or generate a token ID in the Token Credential section by clicking the Add button and provide credentials.
As soon as entities are merged from the feature branch to the master branch in the GitHub repository mentioned in Jenkins, the Import Utility is executed.
On the Build tab, select the Pipeline Script command and create the pipeline (explained in the above section) to execute the Utility.
Click Save.

GitLab settings to integrate with Jenkins

Webhooks trigger the Jenkins job to execute the Import Utility if the defined event happens.

To integrate GitLab with Jenkins, perform the following steps.

In your GitLab Account, select the repository that you want to integrate with Jenkins and go to the Settings tab.
Select the Webhooks option from the left pane.
Click the Add new webhook option.
In the URL field, enter the Jenkins URL. You must add /generic-webhook-trigger/invoke to this and provide secret token credential ID or that you have generated in jenkins while configuring webhooks, as shown in the figure below.
Select the Pushes in the events and click the Add Webhook option.
Select event Push wildcard expression that mentions the main branch name.
Click the Save changes option to apply the changes. Then, click on the Test option to test the webhook.