As a part of the constant endeavor to enhance the ease of working with Kyvos, the Continuous Integration and Continuous Deployment (CI/CD) Utility is created to automate the versioning of Kyvos entities like Semantic Models, Relationships, Datasets, etc., using Git as a versioning tool. It can be used to migrate entities from one Kyvos deployment (Source) to another Kyvos deployment (Target).
...
Parameter | Description |
EXPORT_SEMANTIC_MODELS | Defines whether to export Semantic Model semantic model entities or not.
|
SEMANTIC_MODEL_DETAILS | Provide Provides details of the semantic model that needs to be exported in the following format. : "FolderName":["SemanticModelName"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "SemanticModelName": Comma separated entity names existing within the folder Example {"Folder1": [ “Model1”,” Model2”], “Folder2”: [“Model3”]} |
EXPORT_DATASET_RELATIONSHIPS
| Defines whether to export Dataset Relationship entities or not.
|
DATASET_RELATIONSHIP_DETAILS
| Provide Provides details of the Dataset Relationship , which that must be exported in the following format. "FolderName":["DatasetRelationshipName"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "DatasetRelationshipName": Comma separated entity names existing within the folder Example For example, {"Folder1": [ “DRelationship1”,” DRelationship2”], “Folder2”: [“DRelationship3”]} |
EXPORT_DATASETS | Defines whether to export Dataset entities or not.
|
DATASET_DETAILS | Provide Provides details of the Dataset , which that must be exported in the following format. "FolderName": ["DatasetName"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "DatasetName": Comma separated entity names existing within the folder Example {"Folder1": [ “Dataset 1”,” Dataset 2”], “Folder2”: [“Dataset 3”]} |
EXPORT_WB | Defines whether to export Workbooks or not.
|
WB_DETAILS | Provide Provides details of the Workbooks that must be exported in the following format. "FolderName": ["Workbook"], “FolderName” should be the name of the Kyvos entity folder name to be exported. "Workbook": Comma separated entity names existing within the folder Example For example, {"Folder1": [ “workbook1”,” workbook2”], “Folder2”: [“workbook3”]} |
GIT_PUSH | Defines whether to push the exported entities into Gitlab GitLab or not.
|
GIT_URL | Server URL |
GIT_LOCAL_REPO_PATH | Provide Provides a folder path to be used as a staging location. For example, “C:/CICDUtility/Export/GITRepo” NoteNOTE: Please use Use the forward-slash (/) in the repo path. |
GIT_FEATURE_BRANCH_NAME | Gitlab Specifies the GitLab feature branch name where entities will get exported. |
GIT_FEATURE_BRANCH_CLEAN | Defines whether to clean the GIT Feature branch before a fresh export. The default value is N.
|
GIT_REPONAME | Gitlab Specifies the GitLab repository name where the entities will get exported. |
GIT_USERNAME | Gitlab username Specifies the GitLab user name to connect with GIT Repo. |
GIT_TOKEN | The Personal Access Token for Authentication authentication with the user’s GIT account. If you do not have the Personal Access token, you can create it using the steps explained in the Append Appendix. |
...
Example configurations for different scenarios
...
The parameters are optional and can be set in the serverDetailExportUtility.properties file. If values are given in both places, the script arguments take precedence.
...
Step 1: On changes in the configured Gitlab branch, the Jenkins pipeline triggers to pull the changed files in the local file system; it then launches the ImportKyvosEntities.sh script.
A Jenkins pipeline is required to automate
the identification of the change in the configured Gitlab branch and
pull the changed entity files into a configured folder within the local file system
to launch the ImportKyvosEntities.sh script
Step 2: On execution, the ImportKyvosEntities.shscript reads all the entities xml files placed in the configured folder and uses Kyvos REST APIs to import them in target Kyvos deployment.
| Anchor | ||||||
|---|---|---|---|---|---|---|
|
If the Jenkins pipeline is not configured, Step 2 can still work on launching launch ImportKyvosEntities.sh independently.
Prerequisites
Git Bash: It is a package that installs Bash, some common bash utilities, and Git on an operating system. This is needed to execute the Git commands on your system.
GitLab: Git server with main and feature branches.
Jenkins: For automated execution of the Import utility script, based on a trigger from GitlabGitLab.
Setting up Import Utility
Extract the setup zip.
Configure source target Kyvos deployment details in the serverDetailImportUtility.properties file in the Conf folder. This file needs the details below to be filled in. If the details are not provided in this file, then alternatively, these details can be provided as an argument to the import utility script.
KYVOS_URL = <YourKyvosURL>/rest/
USERNAME = <KyvosUserName>
PASSWORD = <KyvosUserPassword>
Provide the execute Execute the chmod +x ImportKyvosEntities.sh permission on the ImportKyvosEntities.sh file, if not already provided (chmod +x ImportKyvosEntities. sh)
Executing Import Utility separately without Jenkins
Execute the following Import Utility separately without Jenkins to perform step 2.
| Code Block |
|---|
Run ./ImportKyvosEntities.sh $KyvosEntitiesPath $Username $Password $KyvosUrl |
KyvosEntitiesPath is a local machine’s GitLab Repo folder path, storing entities on the pull. This is a mandatory parameter. The other parameters are optional if set in the serverDetailImportUtility.properties file. If values are given at both places, script arguments take the preference.
You can access the execution log file in the "conf/log" folder, which contains the file named " cicd.log.”
Jenkins job setup and integration with GitLab Repository
If the Jenkins job is not created and configured, see the Creating new Project in Jenkinssection.
If the Jenkins job is not Integrated with the GitLab Repository, see the Gitlab GitLab settings to integrate with Jenkinssection.
Once the above steps are completed, you can execute the Import Utility via Jenkins.
...
Kyvos Monitor Image displays updated audit logs for imported Datasets, Relationships, and Semantic Model entities post Import utility execution.
Appendix
| Anchor | ||||||
|---|---|---|---|---|---|---|
|
Creating Personal Access Token in GitLab
...
In your GitLab account, navigate to Settings.
In the left pane, click the Developer settings >Personal access tokens. The Personal access tokens section is displayed. Click Generate new token.
Specify the token a Descriptive name, select the Scopes which defines the access for the Personal tokens, and then click the Generate token. A token is generated. Make sure you
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Note |
...
that will be used in Export Utility. |
...
You can also Update, Regenerate, Revoke, or Delete the token using the options available on the Personal access token page.
Creating new Project in Jenkins
| Anchor | ||||||
|---|---|---|---|---|---|---|
|
To create a new project in Jenkins, perform the following steps.
Login to your Jenkins account.
Install the following plugins:
Git client plugin
GitHub API
GitHub Branch Source Plugin
GitHub plugin
Genric Webhook plugin
If the plugins are not installed while initializing Jenkins, go to Manage Jenkins >Manage Plugins and install the plugins.
Click New Item from the top left of your dashboardDashboard.
Enter the name of the item you want to create, select Pipeline,and click OK Ok. The project is created.
Navigate to the newly created project and select the Configure option, as shown in the following figure.
On the General tab, select the GitHub Project check box and provide the URL of the GitHub/GitLab repository in the Project url field, as shown in the following figure.
On the Build Triggers tab, select Generic Webhook Trigger. This triggers a Jenkins job when the GitLab webhook relays any payload. If you are using a generic webhook, you can select the generic webhook trigger option.
Provide Jenkins credential id ID in the Token section or generate a token id ID in the Token Credential section by clicking on ADD the Add button and provide credentials.
It is used to execute the Import Utility as soon As soon as entities are merged from the Feature feature branch to the Master master branch in the GitHub repository mentioned in Jenkins above, the Import Utility is executed.
On the Build tab, select the Pipeline Script command and create the pipeline (explained in the above section) to execute the Utility.
Click the Savebutton to save your changes.
GitLab settings to integrate with Jenkins
| Anchor | ||
|---|---|---|
|
...
|
Webhooks trigger the Jenkins job to execute the Import Utility if the defined event happens.
...
In your GitLab Account, select the repository that you want to integrate with Jenkins and go to the Settings tab.
Select the Webhooks option from the left pane.
Click the Add new webhook option.
In the URL field, enter the Jenkins URL. You must add /generic-webhook-trigger/invoke to this and provide secret token credential id ID or that you have generated in jenkins while configuring webhooks, as shown in the figure below.
Select the Pushes in the events and click the Add Webhook option.
Select event Push wildcard expression in that mention mentions the main branch name.
Click the Save changes option to apply the changes. Then, click on the Test option to test the webhook.