Kyvos Manager, which is a cluster management tool for Kyvos, currently supports configuring multiple instances of services like BI Server, Query Engine, Kyvos Web Portal, and Postgres that work in high availability mode. Kyvos services can operate normally even in the absence of the Kyvos Manager. However, certain features such as Services HA, configuring properties, Postgres High Availability for bundled Postgres, Hadoop connection Configuration & Lib Sync from Compute cluster or DFS, regardless of SSH or non-SSH mode, are dependent on Kyvos Manager. If any of these features are required at any time, an instance of Kyvos Manager must be running for Kyvos to work properly. Therefore, from Kyvos 2024.1 onwards, Kyvos Manager supports high availability in case of instance failure, and immediately, another instance will be active to serve the requirement.
Prerequisites for using Kyvos Manager High Availability (HA)
Adding Kyvos Manager (KM) instances is supported only on nodes with Kyvos Web Portal, as a cost optimization strategy. Thus, Kyvos Manager HA requires Web Portal HA as a precondition.
For Kyvos Manager HA, the Zookeeper must be up and running.
Each node in a cloud cluster must have the respective CLI command (aws, azcopy, gsutil), while on-premises, it must have the HDFS command and access to HDFS.
For cloud environments, Load balancer must be configured in the cluster.
Irrespective of the type of repository used for Kyvos Manager, the repository must be accessible from all Kyvos Manager nodes.
All newly added instances of Kyvos Manager must have the same permissions (IAM Role, Managed Identity) and network access permissions (Security Group, Local Firewall) as the Kyvos Manager node.
To enable the Kyvos Manager High Availability mode, you need to navigate to the Kyvos Manager Settings > Kyvos Manager Properties. On the Kyvos Manager Properties page, search for ENABLE_KM_HA property, and then set the value of the ENABLE_KM_HA property to 'yes'. By default, this property is set to 'no', which means that the Kyvos Manager HA mode is disabled and will not be displayed on the Kyvos Manager Dashboard. Therefore, if you want to make the Kyvos Manager HA available, you need to update the value of this property to 'yes'. After enabling the Kyvos Manager High Availability through this property, Kyvos Manager will restart. After restarting Kyvos Manager, the Kyvos Manager Disaster Recovery script must be pushed, as shown below.
Important
If Zookeeper goes down, then Kyvos Manager HA will not work till the time Zookeeper gets restored.
If Zookeeper is unavailable then any activities expected from non active Kyvos Manager instance will not work.
If Zookeeper is not available as HA then restore Zookeeper for proper working of Kyvos Manager HA.
In the absence of Zookeeper, all the existing Kyvos Manager instances will continue their last state i.e. active one will remain active and passive instance will continue to remain as passive.
If Kyvos Manager repository is inaccessible or down then Kyvos Manager or Kyvos Manager HA will not work. In this case, you need to perform Disaster Recovery.
If all BI server nodes go down due to scheduling and have Zookeeper deployed on them, Kyvos Manager HA will not work.
It is recommended to have multiple Zookeeper for proper working of Kyvos Manager HA. The cluster having single Zookeeper on Kyvos Manager node and Kyvos Manager node itself becomes down then passive services of Kyvos Manager cannot promote themselves as Active.
Tip
In cloud-based environments, where load balancer is used, all requests will go to active Kyvos Manager only. When a passive Kyvos Manager node URL is accessed, you will be redirected to the load balancer URL or Active Kyvos Manager node URL (when load balancer is not configured).
The Kyvos Managers Nodes section is displayed on the Dashboard, as shown in the below image.
The Kyvos Manager nodes that you have added will be displayed at:
On the Kyvos Manager Dashboard, in the Kyvos Manager Nodes section.
On the Nodes and Services page, and then click the Nodes and Services mapping tab. The Kyvos Manager Nodes section is displayed.
To learn how to add Kyvos Manager nodes, refer to the https://kyvosdocumentation.atlassian.net/wiki/spaces/KD20241/pages/45352414/Adding+and+deleting+nodes#addingKMservice section.
Note
In the Mode column, the Active mode indicates that Kyvos Manager service is currently in use. Conversely, the Passive mode indicates that Kyvos Manager service has been added, but it is currently not in use.
Only one instance of Kyvos Manager service will be active at any given time.
If Kyvos Manager HA is enabled but no additional Kyvos Manager service is added, the default Kyvos Manager instance will be displayed in Kyvos Manager Nodes section on the Kyvos Manager Dashboard.
If the Active Kyvos Manager service becomes passive, the other added service will become active automatically.
Kyvos Manager instance properties
You can view the detailed Kyvos Manager instance properties for both Active and Passive mode. You can also modify the alias name of the node at any time.
To view the properties, perform the following steps.
In the Kyvos Manager Nodes section, click the circle (Red/Green), the Action menu (…) appears.
Click Properties. The Properties dialog is displayed.
On the Properties dialog, you can specify an Alias name to a node which can be easily modified later.
The Alias name will be displayed in the Alias column.
Start, Restart, or Stop Kyvos Manager Service for passive instance
You can start, restart, or stop single Kyvos Manager Service for the service which is in a Passive mode, as shown in below image.
Note
From this section, you cannot start, stop, or restart Kyvos Manager Service that is in Active mode.
Click Start Kyvos Manager Service, the following dialog is displayed.
Provide your Kyvos Manager password and then click Save. The Kyvos Manager Service will start.
Similarly, you can Restart Kyvos Manager Service or Stop Kyvos Manager Service.
Start, Restart, or Stop all passive Kyvos Manager Services
You can start, restart, or stop all Kyvos Manager Services for the services which are in a Passive mode, as shown in the image below.
Click Start Kyvos Manager Service, the following dialog is displayed.
Provide your Kyvos Manager password and then click Save. The Kyvos Manager Service will start.
Similarly, you can Restart Kyvos Manager Service or Stop Kyvos Manager Service.
Note
The Kyvos Manager service in Active mode is always disabled since only passive services can be started, stopped or restarted.
Restart Active Kyvos Manager Instance
You can also restart active Kyvos Manager instance. To do this, click the Action Menu appears at the right of the page ,and then click Restart Active Kyvos Manager Instance.
To restart single Active Kyvos Manager Node, click the Actions (…) next to Kyvos Manager and then click Restart Active Kyvos Manager Instance, as shown in below image.
The following dialog is displayed.
Provide your Kyvos Manager password and then click Save. The Kyvos Manager Service will restart.
Behavior Change
If Kyvos Manager HA is not configured:
Default existing single Kyvos Manager node will continue to work as an Active node.
On Kyvos Manager Dashboard, no rows related to where Kyvos Manager instances are running will be displayed.
On the Migrate role page, no option to add or remove Kyvos Manager instance will be available.
Snapshots will not be pushed in ON PREM deployments during Upgrade, Rollback, Patch.
Kyvos Manager will not connect to Zookeeper.
Start, stop, or restart Kyvos Manager server will continue to appear as it is like earlier
On changing the Kyvos Manager property, only Kyvos Manager configuration bundle snapshot was pushed. However, if Kyvos Manager HA property is enabled, then all the Kyvos Manager snapshot bundles will be pushed.
Schema for Kyvos Manager HA instances will be created even if Kyvos Manager HA is not enabled.
Known limitation
As this is a initial version of Kyvos Manager HA for the Kyvos 2024.1 release, you will encounter the following limitations:
Download logs of passive Kyvos Manager instances are not supported.
Make a passive Kyvos Manager instance as an active instance from Kyvos Manager is not supported.
Configuring Kyvos Manager HA during a cluster deployment (irrespective of whether automated or wizard based) is not supported.