Versions Compared

Old Version 1

changes.mady.by.user Sarabjeet Kaur

Saved on

compared with

New Version 2

changes.mady.by.user Sarabjeet Kaur

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Applies to: (tick) Kyvos Enterprise  (tick) Kyvos Cloud (SaaS on AWS) (tick) Kyvos AWS Marketplace

...

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

  • Keep the following things handy during disaster recovery, depending on what is affected in your cluster.

    • New certificates are applicable if existing settings (domain/subdomain) are changed after recovery.

    • Production license as per new BI nodes in case any BI server impacted

  • You must use the disaster recovery capability in any of the following scenarios: 

    • If Kyvos Manager, BI Server, or Query Engine nodes are affected. 

    • If only the Kyvos Manager nodes are affected. 

    • If Kyvos Manager and all Kyvos nodes (BI Servers, Query Engines, WebPortalWeb Portal, and Postgres Server) are affected. 

  • If only the BI Server or Query Engine nodes are affected, then add a node for that service, and the cluster can be restored. You will not need to use disaster recovery capability for this case.

  • If you enable TLS for Kyvos Manager and Kyvos application, the TLS option is not applicable during the Disaster Recovery restoration. After successful restoration, the TLS-related certificates are restored, and you can continue with the TLS option. 

Disaster recovery through the guided flow on Kyvos Manager

...

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

Extract the KyvosManager Kyvos Manager DB snapshot tar by keeping it parallel to the kyvosmanagerdata folder. This will ensure that after untar of km_db_snapshot.tar.gz, the ankushdb folder is created at the kyvosmanagerdata/server/db/ location.

  1. Start Kyvos Manager using the startup.sh script.

  2. On the Kyvos Manager, navigate to the Kyvos Manager > Settings, and perform the following steps.

    1. In the Kyvos Manager Server Details area, click Reconfigure.

    2. Update the Hostname and Port for Kyvos Manager.

    3. Click the Validate button. You will see a validation error Server accessibility failed from 1 node. This is due to the unavailability of the old Kyvos Manager node.

    4. Click Apply.

  3. Navigate to the Dashboard. The cluster dashboard will show Unable to get license info error (see image below). Ignore it till a new KM node gets added to the cluster.

  4. Stop Kyvos component services using the Actions menu for each component.

  5. Click Manage Kyvos > Disaster Recovery on the navigation pane.
    Depending on the current state of the system, you may see up to 3 links.

    1. If Kyvos Manager-managed multi-node Zookeeper was deployed, then the first link will appear for Zookeeper removal. For a single-node Kyvos Manager-managed zookeeper, no such link will appear.

    2. Then, you will see a link for removing the old Kyvos Manager node, which is no longer available.

    3. Thereafter, you will see a link to Add a new (current) Kyvos Manager node.

      Warning

      You MUST click the links in the same order as they are listed (i.e., first remove the Zookeeper (if applicable), then remove the unreachable node, and finally add the new node.

  6. Remove Zookeeper using the link.

  7. Remove the old Kyvos Manager node using the Remove Unreachable Node link. This initiates the Remove Node operation for removing the node having a WebPortal role (and Postgres role if bundled Repository is being used) from the cluster. You will be redirected to the Remove Node operation details page.

  8. Go to the Disaster Recovery page, and perform the following steps.

    1. Click the Add Node link to add a new Kyvos Manager node to the cluster. This will initiate the Add Node operation for adding the Web Portal role on the new Kyvos Manager node. You will be redirected to the Add Node operation details page.

    2. In case of any failure, re-perform this operation.

    3. On successful completion of this operation, the Kyvos folder will be added to this node.

  9. If bundled Postgres was in use, then:

    1. Download the Postgres snapshot bundle from binaries, and by deleting the existing Postgres folder on the KM node, set up the extracted folder from this snapshot as Postgres on the node. This Postgres snapshot needs to be extracted by copying it in parallel to the kyvos foler.

    2. Download the latest/applicable Postgres dump bundle from DFS (from the data folder) to the new Kyvos Manager node.

    3. Start Postgres service on the Kyvos Manager node.

    4. Import the dump in the Postgres instance (see the Manage Kyvos Repository section)

  10. On the Switch Repository page, configure the bundled repository on the Kyvos Manager node (see the Manage Kyvos Repository section).

  11. If any additional nodes are impacted, then:

    1. Remove those nodes using the Delete Node functionality of Kyvos Manager.

    2. Add the newly created node with the required roles on it.

  12. For cloud-based clusters, add Zookeeper to the cluster depending on how Zookeeper was used earlier.

    1. If non managed Zookeeper was in use, then configure a new KM node ip:2181 as a value for the Zookeeper string from the Hadoop Ecosystem configuration page.

    2. If Kyvos Manager-managed Zookeeper was in use, then deploy the Zookeeper component from the Hadoop Ecosystem configuration page.

  13. Start Kyvos Component services from the Dashboard using the Actions menu.

Panel
panelIconIdatlassian-info
panelIcon:info:
bgColor#FFFAE6

Important

When Kyvos Manager HA is enabled and Managed zookeeper is used then after completing the Disaster Recovery activity, restart Kyvos Manager irrespective of whether TLS is enabled or not. Prior to the Kyvos 2024.1 release, Kyvos Manager restart is required only when TLS is enabled.

Checkpoints

Some important checkpoints that you must verify after completing the disaster recovery process.

...

If the Kyvos Manager node is restored using an image or disk snapshot, then ensure that:

  • Kyvosmanager Kyvos Manager agent process running on a node is stopped

  • Crontab entry for the agent is removed

  • Folders kyvos, kyvosmanagerdata, and Kyvos_States_Backup parallel to it are also deleted before adding this new Kyvos Manager node to the cluster.

...