Managing Postgres as Repository for Kyvos Manager

Applies to: Kyvos Enterprise  Kyvos Cloud (Managed Services on AWS)   Kyvos Azure Marketplace

Kyvos AWS Marketplace  Kyvos Single Node Installation (Kyvos SNI)   Kyvos Free ( Limited offering for AWS)

New Deployment 

Prerequisites 

• Using External Repository as Kyvos Manager Repository 

Whether you deploy a new Kyvos cluster using the wizard-based deployment or migrate an existing Kyvos Manager database, you will need to use an external repository as the Kyvos Manager repository. In either case, the external repository must meet the following requirements.

1. 1. The Database server/instance must be up and accessible from the Kyvos Manager node over the port on which the database service is serving requests over JDBC.

   2. For network-level access (Security Group), the same set of permissions/roles that are required for the external database in Kyvos should be applied. To allow access from the Kyvos Manager node, the port in the external repository security group needs to be opened.
      NOTE: No new additional permissions or port access is required. 

   3. The database name configured as the Kyvos Manager repository must exist in the configured database instance.

   4. The database access you configured for accessing the Kyvos Manager database must exist (pre-created) and have permissions on the database configured for the Kyvos Manager.

   5. If the Kyvos Manager is expected to be used independently (i.e., not set up using automated deployment) and the wizard-based deployment is required before starting the Kyvos Manager, the following properties must be configured in the jdbc.properties file. 

      (jdbc.url=, jdbc.username=, jdbc.password=, useBundledRepo=false, cloudProvider=AWS|AZURE|GCP, repoIdentifier= ,projectName=, serviceAccountName=) some of which like projectName & serviceAccountName are applicable only for a particular cloud provider.
      

• Using Bundled Postgres High Availability

The bundled Postgres host (which is, by default, the Kyvos Manager node) must be accessible from the Kyvos Manager node. In a cloud-based environment, ensure that the Kyvos Manager can access port 45421 on any node of the cluster if the bundled Postgres is moved to another node. Therefore, the security group added to other nodes must allow access to port 45421 from any node of the cluster.

Creating resources 

• Using Automated deployment 
  Kyvos supports bundled and external repositories for both AWS and Azure.

  • For AWS 
    • To create Kyvos resources for creating an AWS CloudFormation template for use in automated deployment with EMR, see the Creating Kyvos resources for AWS automated deployment with EMR section for more details.
    • To create Kyvos resources for creating an AWS CloudFormation template for use in automated deployment with Databricks, see the Creating Kyvos resources for AWS automated deployment with Databricks section for more details.
  • For Azure
    • To create Kyvos resources for creating an ARM template, see the Creating Kyvos resources for Azure through Kyvos Manager and deploying cluster section for more details. 
  • For GCP
    • Only the Bundled Postgres repository is supported by default. See the Creating Kyvos Resources for GCP using Script section for more details. 

• Using Wizard-based deployment 

To configure the Kyvos Manager repository for the first time using the wizard-based deployment (any platform) to use an external repository, do the following:

• • To use any existing Postgres repository instead of the bundled Postgres, you must update the jdbc.properties file in kyvosmanagerdata/server/repo/db/ folder before starting Kyvos Manager. In this file, you need to update the details of the applicable attributes, such as jdbc.url, jdbc.username, jdbc.password, and secret name (if applicable).
  • Some attributes may be specific to the external repository, while others may be specific to the cloud if a cloud-based external repository is intended to be used.
    Refer to Step-e in the Using External Repository as Kyvos Manager Repository section. 

Existing Deployment

This section is applicable when you are migrating from Derby to Postgres as a repository for Kyvos Manager. 

Behavior changes 

• User experience for specific operations
• • Rollback: The start of the rollback operation will take some time.
  • Migrate Role: After migrating from Derby to Postgres, Kyvos Manager will restart automatically. You are prompted to log in to Kyvos Manager. 
  • Postgres High Availability: After configuring Postgres HA, Kyvos Manager will restart automatically. You are prompted to log in to Kyvos Manager. 

  • TLS enable or disable: For bundled Kyvos Manager repository, restarting Kyvos & KyvosManager is mandatory. In the case of the external Kyvos Manager repository, you need to manually restart Kyvos and Kyvos Manager. 

• Rollback 

From Kyvos 2023.1 release and onwards, if you roll back to a Kyvos Manager build that supports Derby as its repository, any operations, such as audits, events, or other activities (like users add, import, delete, and update) performed using Kyvos Manager build 2023.1 and onwards will not be available when you rollback to a Kyvos Manager release previous to 2023.1.

See the Updating Application Version section for more details. 

• Restoring to Derby State of Kyvos Manager 

You can restore the Derby-compatible version of the Kyvos Manager build that is prior to Kyvos 2023.1. See the Updating Application Version section for more details. 

You must perform the following manual steps to perform a rollback for Kyvos or Kyvos Manager build:

1. Stop Kyvos services.
2. Stop the current Postgres-based Kyvos Manager if already running (using the stop-km.sh script).

3. Stop the Postgres server running on the applicable node (located at the path parallel to the installed Kyvos).
   

   Info

   This step is applicable only if KyvosManager is using bundled Postgres repository. 

4. Go to the Kyvos path. For example, /data/kyvos/app/kyvos/ and perform the unlink postgres command, which will unlink/remove the postgres soft link. 
5. Move the Postgres folder inside kyvos on all the nodes of the cluster (if it exists).
   This is also applicable only if bundled Postgres is used as the Kyvos Manager repository. 
6. Change jdbc.properties.
   NOTE: The files are located at:  kyvosmanagerdata/server/db/
   

7. Start old Kyvos Manager (derby supported) using the startup.sh script. 
   
   

   Info

   If Kyvos rollback is required, you can perform it using the derby-supported Kyvos Manager build. 

• Specifying node for running Postgres Service

When the Kyvos Manager uses bundled Postgres, you cannot specify a node other than the Kyvos Manager node to run the bundled Postgres service for Kyvos. Currently, by default, the Kyvos Manager node is also used as the node for running the bundled Postgres service. However, it is user-configurable, so you can change it.

Upgrading

You should take a manual backup before upgrading Kyvo Manager from Derby to Postgres.  

Manual backup of Kyvos

When bundled PostgreSQL is used as the repository in KyvosManager, the PostgreSQL folder is moved out of Kyvos and placed parallel to it on all the cluster nodes. 

If you take a manual backup of Kyvos, ensure that you also take the backup of the compatible state of the PostgreSQL folder that is located parallel to the Kyvos folder.

Auto Migration of Data from Derby to Postgres

When the Kyvos Manager, built only with Postgres as its repository, is started with Derby configured as the Kyvos Manager repository, you will be automatically redirected to the page for database migration after logging in to the Kyvos Manager. Using this page, you can migrate data from the existing Derby database to a new Postgres database that you specify (either bundled or external). 

See the Updating Application Version section for more details. 

To auto migrate Data from Derby to Postgres, perform the following steps: 

1. After upgrading Kyvos Manager, the Export Data and Backup wizard is displayed. 
   
2. Click Start Data Export.
   Kyvos Manager creates a data backup while exporting data from the Apache Derby database. 
   
3. Click Next. You need to validate the Kyvos Manager repository connection. 
   
4. Click Test Connection to validate the Database node where you want to migrate your data (Derby-compatible Kyvos Manager).
   

5. Click Migrate Now. 
   

   Info

   Kyvos Manager will restart multiple times during repository configuration and data import. You are prompted to log in to Kyvos Manager during the operations. 

6. Once the Repository configuration is completed and the Data is imported, click Finish. You can view the status details on the Operations page of the Kyvos Manager. 

Creating External Repository for Kyvos Manager 

If you want to create an external repository for Kyvos Manager, download the Cloud Formation Scripts.

• Script for AWS: Click here to download
• Script for Azure: Click here to download

Permissions

Before using PostgreSQL as the Kyvos Manager repository, ensure the following: 

• No additional permissions are required for using PostgreSQL as the Kyvos Manager repository. However, if the permission roles created earlier did not have the necessary permissions related to the external repository. Those permissions must be configured in the client environment before accessing the external repository.
• You must have access to the external PostgreSQL Kyvos Manager repository over its applicable port.
• For automated deployment, JDBC is already configured as the Kyvos Manager repository; hence no need to discover an external Postgres instance.
• For bundled PostgreSQL as the KyvosManager repository, the JDBC configuration is automated during the wizard-based deployment. However, for an external PostgreSQL as the KyvosManager repository, you need to manually configure JDBC before starting KyvosManager.
• No additional permission is required for migration (in case of switching from Derby to Postgres in the existing environments). The permission to fetch external repository details in the migrate UI does not require any additional permission, as the same repository fetching functionality already exists in the switch repository feature provided by KyvosManager for switching the Kyvos repository from bundled Postgres to external Postgres. 

Scripts to Manually Start or Stop Kyvos Manager

From the Kyvos 2023.1 release, two new scripts, start-km.sh  and  stop-km.sh  will be used to manually start and stop Kyvos Manager, respectively. You can also continue to use the existing scripts to start and stop the Kyvos Manager services. However, if you have Kyvos Manager High Availability configured, you must use the new scripts. Both new  and old scripts are available in the  /data/kyvos/installs/kyvosmanager_war/kyvosmanager/bin/  folder. 

Managing Bundled Postgress Service 

When using the bundled Postgres as the Kyvos Manager repository, you will observe the following while starting and stopping the Postgres service:

1. The Postgres service will not be involved during the start, stop, and restart of the Kyvos cluster and Kyvos component-level services. As a result, the management options (Start/Stop/Restart) for the Postgres role will not be available on the Cluster Dashboard.
2. There will be a change in the sequence for stopping/starting the bundled Postgres service, where it is required to restart bundled Postgres.