Kyvos Manager High Availability

Kyvos Manager, a cluster management tool for Kyvos, currently supports configuring multiple instances of services like BI Server, Query Engine, Kyvos Web Portal, and Postgres that work in high availability mode. Kyvos services can operate normally even in the absence of the Kyvos Manager. However, certain features such as Services HA, configuring properties, Postgres High Availability for bundled Postgres, Hadoop connection Configuration & Lib Sync from Compute cluster or DFS, regardless of SSH or non-SSH mode, are dependent on Kyvos Manager. If any of these features are required at any time, an instance of Kyvos Manager must be running for Kyvos to work correctly. Therefore, from Kyvos 2024.1 onwards, Kyvos Manager supports high availability in case of instance failure, and immediately, another instance will be active to serve the requirement.

Prerequisites for using Kyvos Manager High Availability (HA)

The cluster deployed with earlier releases, before adding or removing Kyvos Manager Service, ensure that the permissions required for Load Balancer entries manipulation are added to the role.
See the Permissions for Load Balancer entries manipulation section.
Adding Kyvos Manager (KM) instances is supported only on nodes with Kyvos Web Portal as a cost optimization strategy. Thus, Kyvos Manager HA requires Web Portal HA as a precondition.
For Kyvos Manager HA, the Zookeeper must be up and running.
Each node in a cloud cluster must have the respective CLI command (aws, azcopy, gsutil), while on-premises, it must have the HDFS command and access to HDFS.
For cloud environments, a Load balancer must be configured in the cluster. If the Load balancer is not configured, you need to provide its details.
- For AWS, the Load Balancer needs to be from the same region in which other resources of the Kyvos cluster are deployed.
- For Azure, the Load Balancer needs to be from the same resource group and region in which other resources of the Kyvos cluster are deployed.
- For GCP, the Load Balancer needs to be in the same project and region in which other resources of the Kyvos cluster are deployed.
Irrespective of the type of repository used for Kyvos Manager, the repository must be accessible from all Kyvos Manager nodes.
All newly added instances of Kyvos Manager must have the same permissions (IAM Role, Managed Identity) and network access permissions (Security Group, Local Firewall) as the Kyvos Manager node.
To enable the Kyvos Manager High Availability mode, navigate to the Kyvos Manager Settings > Kyvos Manager Properties. On the Kyvos Manager Properties page, search for the ENABLE_KM_HA property, and then set the value of the ENABLE_KM_HA property to ‘yes.’ By default, this property is set to ‘no,’ which means that the Kyvos Manager HA mode is disabled and will not be displayed on the Kyvos Manager Dashboard. Therefore, if you want to make the Kyvos Manager HA available, you need to update the value of this property to ‘yes.’ After enabling the Kyvos Manager High Availability through this property, Kyvos Manager will restart. After restarting Kyvos Manager, the Kyvos Manager Disaster Recovery script must be pushed, as shown below.

Important

If Zookeeper goes down, then Kyvos Manager HA will not work till the time Zookeeper gets restored.
If Zookeeper is unavailable, any activities expected from non active Kyvos Manager instance will not work.
If Zookeeper is unavailable as HA, restore Zookeeper for the proper working of Kyvos Manager HA.
In the absence of Zookeeper, all the existing Kyvos Manager instances will continue their last state, i.e., the active one will remain active, and the passive instance will continue to remain passive.
If the Kyvos Manager repository is inaccessible or down, then Kyvos Manager or Kyvos Manager HA will not work. In this case, you need to perform Disaster Recovery.
If all BI server nodes go down due to scheduling and Zookeeper is deployed, Kyvos Manager HA will not work.
It is recommended to have multiple Zookeepers for the proper working of Kyvos Manager HA. The cluster has a single Zookeeper on the Kyvos Manager node, and the Kyvos Manager node itself becomes down, so passive services of Kyvos Manager cannot promote themselves as Active.

Tip

In cloud-based environments, where a load balancer is used, requests will only go to the active Kyvos Manager. When a passive Kyvos Manager node URL is accessed, you will be redirected to the load balancer URL or Active Kyvos Manager node URL (when the load balancer is not configured).

The Kyvos Managers Services section is displayed on the Dashboard, as shown in the below image.

The Kyvos Manager services that you have added will be displayed at:

On the Kyvos Manager Dashboard, in the Kyvos Manager Services section.
On the Nodes and Services page, and then click the Nodes and Services mapping tab. The Kyvos Manager Services section is displayed.

To learn how to add Kyvos Manager nodes, refer to the Managing Services Mapping and Nodes | addingKMservice section.

Note

In the Mode column, the Active mode indicates that the Kyvos Manager service is currently in use. Conversely, the Passive mode indicates that the Kyvos Manager service has been added but is currently unavailable.
Only one instance of the Kyvos Manager service will be active at any given time.
If Kyvos Manager HA is enabled but no additional Kyvos Manager service is added, the default Kyvos Manager instance will be displayed in the Kyvos Manager Nodes section on the Kyvos Manager Dashboard.
If the Active Kyvos Manager service becomes passive, the other added service will become active automatically.

Kyvos Manager instance properties

You can view the detailed Kyvos Manager instance properties for Active and Passive mode. You can also modify the alias name of the node at any time.

To view the properties, perform the following steps.

In the Kyvos Manager Services section, click the circle (Red/Green), and the Action menu (…) appears.
Click Properties. The Properties dialog is displayed.
On the Properties dialog, you can specify an Alias name for a node, which can be easily modified later.
The Alias name will be displayed in the Alias column.

Start, Restart, or Stop Kyvos Manager Service for passive instance

You can start, restart, or stop a single Kyvos Manager Service for the service that is in a Passive mode, as shown in the image below.

Click Start Kyvos Manager Service, and the following dialog is displayed.
- Provide your Kyvos Manager password and then click Save. The Kyvos Manager Service will start.

Similarly, you can Restart Kyvos Manager Service or Stop Kyvos Manager Service.

Start, Restart, or Stop all passive Kyvos Manager Services

You can start, restart, or stop all Kyvos Manager Services for the services in a Passive mode, as shown in the image below.

Click Start Kyvos Manager Service, and the following dialog is displayed.
- Provide your Kyvos Manager password and then click Save. The Kyvos Manager Service will start.

Similarly, you can Restart Kyvos Manager Service or Stop Kyvos Manager Service.

Restart Active Kyvos Manager Instance

You can also restart the active Kyvos Manager instance. To do this, click the Action Menu at the right of the page, and then click Restart Active Kyvos Manager Instance.

To restart the single Active Kyvos Manager Service, click the Actions (…) next to Kyvos Manager and then click Restart Active Kyvos Manager Service, as shown in the image below.

The following dialog is displayed.

Provide your Kyvos Manager password and then click Save. The Kyvos Manager Service will restart.

Behavior Change

If Kyvos Manager HA is not configured:

The default existing single Kyvos Manager node will continue to work as an Active node.
On Kyvos Manager Dashboard, no rows related to where Kyvos Manager instances are running will be displayed.
On the Migrate role page, no option to add or remove the Kyvos Manager instance will be available.
Snapshots will not be pushed during Upgrade, Rollback, Patch in on-prem deployments.
Kyvos Manager will not connect to Zookeeper.
Start, stop, or restart the Kyvos Manager server, which will continue to appear as it was earlier.
When changing the Kyvos Manager property, only the Kyvos Manager configuration bundle snapshot was pushed. However, if the Kyvos Manager HA property is enabled, all the Kyvos Manager snapshot bundles will be pushed.
Schema for Kyvos Manager HA instances will be created even if Kyvos Manager HA is not enabled.

Known limitation

As this is an initial version of Kyvos Manager HA for the Kyvos 2024.1 release, you will encounter the following limitations:

Download logs of passive Kyvos Manager instances are not supported.
Make a passive Kyvos Manager instance as an active instance from Kyvos Manager is not supported.
Configuring Kyvos Manager HA during a cluster deployment (irrespective of whether automated or wizard-based) is not supported.

Read more: