Content Comparison

As a part of the constant endeavor to enhance the ease of working with Kyvos, the Continuous Integration and Continuous Deployment (CI/CD) Utility is created to automate the versioning of Kyvos entities like Semantic Models, Relationships, Datasets, etc., using Git as a versioning tool. It can be used to migrate entities from one Kyvos deployment (Source) to another Kyvos deployment (Target).

• It uses Gitlab to store versions of Kyvos entities in XML format.  

• It also provides integration with Bitbucket repositories.  

• It is a Java-based utility and that uses Kyvos REST APIs to export the entities from source Kyvos deployment and to import the entities them to target Kyvos deployment. It is designed to be used in Linux systemsystems. 

...

...

The CI/CD Utility comprises the following two utilities:

• Export Utility: It fetches entities from Kyvos deployment in XML format, stores them in a local folder, and then pushes them to the configured Git repository. It has a configuration file to define all entities that are to be fetched along with source Kyvos deployment and git repository details.

• Import Utility: It uses Jenkins to automate the import of entities from the Git repository to target Kyvos deployment.  The Jenkins pipeline is configured in the Gitlab server to pull the entities from Gitlab to the local file system. Jenkins then launches the import utility shell script, which then finally imports the entities from the file system to target Kyvos deployment.

...

The utility bundle contains a single zip file having the following folders and files within it:  

• Conf folder: It contains serverDetailExportUtility.properties, serverDetailImportUtility.properties, and cicdLogs.properties properties 

• Input folder: It contains an input.json file to be used for configuring the entities to be exported and configure the exported entities and the Gitlab server details. 

• Lib folder: This folder containing contains the required libraries for the utility utility. 

• output Output folder: This initially blank folder and to be used for placing will place exported entities' XML file files when not pushing them into the Gitlab server.  

• ExportKyvosEntities.sh: Script This is a script file to launch the Export utility utility.  

• ImportKyvosEntities.sh: Script This is a script file to launch the Import utility utility.  

Export Utility 

Prerequisites 

1. Git Bash: It is a package that installs Bash, some common bash Bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system. 

2. GitLab: Git server with main and feature branches. 

Setting up Export Utility 

1. Extract the setup zip file.  

2. Configure source Kyvos deployment details in the serverDetailExportUtility.properties file located in the Conf folder. This file needs Provide the following details below to be filled in this file. If you don't provide the details are not provided in this the file, then alternatively these details can be provided you can alternatively provide them as an argument to the export utility script. 

   • KYVOS_URL = <YourKyvosURL>/rest/ 

   • USERNAME = <KyvosUserName> 

   • PASSWORD = <KyvosUserPassword> 

3. Configure input.json within the input folder to define the entities to be exported.  The following table provides details of the parameters in the input.json file. 

4. Provide

...

1. execute permission on

...

1. ExportKyvosEntities.sh file,

...

1. using chmod +x ExportKyvosEntities.sh

...

1. , if not already provided.

...

• Export single/multiple entities from ROOT (when entities are not a part of any folder)

"SEMANTIC_MODEL_DETAILS": { "ROOT" : [ “Model4”, “Model5”] }

...

The parameters are optional , if and can be set in the serverDetailExportUtility.properties file. If values are given at in both places, the script arguments take the preferenceprecedence.

You can access the execution log file located within in the "conf/log" folder, which contains the file named " folder containing cicd.log".

Steps Performed performed by Export Utility

1. If the GITPUSH parameter is set as Y, then all the entity XML files will get be exported to the GitLab Feature branch in the following order.

   1. A GIT PULL is performed from the feature branch specified in the input file . So that to sync the remote and local repository (GITRepo) are in sync.

   2. The entities are exported to the GITRepo folder from Kyvos, which is a local repository clone of the GitLab repository.

   3. Entities are then pushed to the GitLab feature branch.

   If

   1. .

2. The entities are directly exported to the Output folder if the GITPUSH parameter is set as N in the input file, then the entities are directly exported to the Output folder. No GIT PULL and GIT PUSH will be performed in this case.

...

The import utility works in two steps.

Step 1: On changes in the configured Gitlab branch, the Jenkins pipeline triggers to pull the changed files in the local file system, ; it then launches the ImportKyvosEntities.sh script.

A Jenkins pipeline is required to automate

1. the changes identification of the change in the configured Gitlab branch and

2. pull the changed entity files in into a configured folder within the local file system

3. to launch the ImportKyvosEntities.sh script

Step 2: On execution, ImportKyvosEntities.sh script reads all the entities xml files placed in the configured folder and uses Kyvos REST APIs to import them in target Kyvos deployment.

In case If the Jenkins pipeline is not configured, then Step 2 can still work on launching ImportKyvosEntities.sh independently.

...

1. Extract the setup zip. 

2. Configure source target Kyvos deployment details in the serverDetailImportUtility.properties file in the Conf folder. This file needs the details below to be filled in. If the details are not provided in this file, then alternatively, these details can be provided as an argument to the import utility script.

   • KYVOS_URL = <YourKyvosURL>/rest/

   • USERNAME = <KyvosUserName>

   • PASSWORD = <KyvosUserPassword>

3. Provide the execute permission on the ImportKyvosEntities.sh file, if not already provided (chmod +x ImportKyvosEntities.sh)

...

Run ./ImportKyvosEntities.sh $KyvosEntitiesPath $Username $Password $KyvosUrl 

KyvosEntitiesPath is a local machine’s GitLab Repo folder path where , storing entities are stored on the pull. This is a mandatory parameter. The other parameters are optional , if set in the serverDetailImportUtility.properties file. If values are given at both places, script arguments take the preference.

You can access the execution log file located within in the "conf/log" folder, which contains the file named "cicd.log".”

Jenkins job setup and integration with GitLab Repository

In case If the Jenkins job is not created and configured, see the section Creating new Project in Jenkins section.

In case If the Jenkins job is not Integrated with the GitLab Repository, see the section Gitlab settings to integrate with Jenkins section.

Once the above steps are completed, you can execute the Import Utility via Jenkins.

...

Trigger the Import utility by executing the following pipeline using the Jenkins job.  Once configured, the job will automatically be triggered by webhook whenever you push entities to the main branch.

sh "jenkinsWorkspace/ImportUtilityFilePath ${KyvosEntitiesPath} ${KyvosUsername} ${KyvosPassword} ${KyvosURL}" 

All the parameters are mandatory here. Add these parameters in to Jenkins pipeline:

• KyvosEntitiesPath: represents the directory path storing Kyvos entities within the Jenkins workspace dedicated to your project. During the Jenkins workflow, data Data is fetched from the main GitLab repository during the Jenkins workflow and stored in this location. Subsequently, this directory serves as is the source for deploying data to the intended environment. It's crucial to run the

...

• KyvosUsername KyvosUsername is the name of the native user with admin access to the target Kyvos deployment.

• KyvosPassword is  is the password of the native user with admin access to the target Kyvos deployment.

• KyvosURL KyvosURL is the URL of the target Kyvos deployment.

Jenkins pipeline

...

pipeline {
    agent any
    stages {
        stage ('Environment Details') {
            steps {
                sh '''
                env | grep -e PATH -e JAVA_HOME
                which java
                java -version
                mvn –version"” }}
             stage('Checkout') {
            steps {
                checkout (
                    [$class: 'GitSCM',
                    branches: [[name: '*/your_branch_name']],
                    extensions: [], userRemoteConfigs:
                    [[credentialsId: 'your_credential_id',
                    url: 'your_git_url']]]   ) } }  
        stage('Build') {
            steps {
                script{
                 
                    dir('jenkinsWorkSpacePath/cicdUtilityPath') {
                        sh "pwd"
                sh " jenkinsWorkSpacePath /cicdUtilityPath/ImportKyvosEntities.sh ${kyvosEntitiesPath} ${KyvosUsername} ${kyvosPassword} ${KyvosURL}"     }  } } } } }

• Kyvos Monitor Image shows displays updated audit entries logs for imported Datasets, RelationshipRelationships, and Semantic Model entities after the execution of post Import utility execution.

  

Appendix

Creating Personal Access Token in GitLab

The Personal Access Token is required for authentication of Export Utility with Git. You are required to provide this in the Export Utility config file, as explained in the Export Utility  sectionsection. To create a Personal Access Token in GitLab, perform the following steps.

...

1. In your GitLab account, navigate to Settings.

2. On In the left pane, click the Developer settings option, select the >Personal access tokens. The Personal access tokens from the left pane, and click section is displayed. Click Generate new token.

   

3. Give Specify the token a Descriptive name, select the Scopes which defines the access for the Personal tokens, and then click the Generate token.

4. A token is generated. Make sure you copy your new personal access token to be used in Export Utility.

   Image RemovedImage Added

5. You can also Update, Regenerate, Revoke, or Delete the token using the options available on the page.

...