Kyvos on AWS - Disaster Recovery and Backup Plan

Applies to:Kyvos Enterprise  Kyvos Cloud (Managed Services on AWS)  Kyvos Azure Marketplace

Kyvos AWS Marketplace  Kyvos Single Node Installation (Kyvos SNI)  Kyvos Free (Limited offering for AWS)

Prerequisites 

1. To create a back up of the data stored in S3 buckets, Kyvos uses the following features of AWS: 
   1. Cross-Region Replication   
   2. Versioning, along with their lifecycle configurations 
   3. Replication configurations. 
2. To back up data, Kyvos needs a separate destination bucket in a region other than that of the source bucket. 
3. Cross account destination bucket replication provisioning should not be provided. 
4. CRR should be enabled on the source bucket only (or its subset). 
5. CRR requires the source and destination buckets with versioning enabled.

Setting up replication bucket

1. Create a destination bucket where you want to set up a Disaster Recovery (DR) cluster.  
2. Enable versioning on both Source and Destination S3 buckets.  
3. Create Lifecycle rules. 
4. On Source bucket enable Replication and perform the following steps: 
   1. Source section 
      1. Provide replication rule name. 
      2. In Source choose a rule Scope as Limit the scope of this rule using one or more filters. 
      3. Provide the prefix as kyvos working directory, such as: user/engine_work 
   2. Destination section 
      1. Choose a bucket in this account. 
      2. Provide IAM roles for copying data from bucket one to two. Refer to the AWS document for details. 
      3. In Additional replication options, select the Delete Marker Replication option. 

NOTE: You must manually copy the existing data after enabling Replication. 

Setting up DR in RDS (Postgres) 

Refer to the AWS document for configuring DR in RDS. 

Kyvos uses manual backup for cross-region replication. The manual backup option is ensuring by our scripts which is insuring by Kyvos scripts. 

Restoring data in case of a disaster 

To restore data in case of a disaster, perform the following steps. 

1. Log in to your AWS account.
2. Select the new region where you have replicated your S3 data. 
3. Go to CloudFormation and deploy the new DR template of Kyvos. 
4. Provide all the required parameters in the template. 
5. Bucket Name should be the same as Replication Bucket. 
6. IAM role of Lambda and EC2 should be the same as original deployment. 
7. Provide Existing RDS, that should be restored before deploying the DR template. 
8. CIDR for VPC creation should same as the original deployment.

Points to remember 

When using this solution, keep the following in mind:  

1. If DR happened, then you cannot move to the Original installation. The DR cluster will be the Primary cluster and you must set up the replication bucket again. 
2. Once the deployment is complete, you MUST change the S3 bucket name in all the datasets, as the raw data bucket is also changed due to DR.
3. If the primary deployment was on a private network (tunneling established between Customer and Kyvos AWS VPC), then you must do the same procedure again after DR deployment. 
4. Once the deployment is complete, you need to wait for Cuboid replication on all the query engines for executing queries. 
5. Once the deployment is complete, you must enable TLS and SSL again in Kyvos. 
6. If any additional IPs were allowed in the Security group of primary installation, you MUST configure the same in the DR Security Group too. 
7. Once the DR deployment is complete, you need to create the custom URL and DNS mapping again. 
8. You need to manage the Glue tables and source data after the DR deployment.

Post DR deployment steps 

After completing the DR deployment, perform the following: 

1. Ensure the license has been uploaded and verified from both Kyvos and Kyvos Manager UI.
   NOTE: You will need a new license will be required as the Kyvos production license is mapped with instance ID. 
2. Verify user can log in to the Kyvos Manager portal. 
3. Verify the desired number of BI Servers and Query Engines are available on Kyvos Manager Portal. 
4. Verify user can log in to Kyvos Portal. 
5. Verify all the users are imported into Kyvos. 
6. Verify access rights of the imported users. 
7. Verify the desired number of BI Servers and Query Engines are up and running. 
8. Verify validation and preview operation on datasets created on S3 files. 
9. Verify validation and preview operation on datasets created on HCatalog. 
10. Verify validation and preview operation on datasets created on Snowflake. 
11. Verify all data has been recovered (check if all entities are available). 
12. Verify job history of entities. 
13. Verify build cubes are browsable. 
14. Verify all the Query Engines can serve queries. 
15. Verify cuboids are replicated locally on Query Engines disks. 
16. Verify only incremented data is processed in incremental builds. 
17. Verify Live query analysis is working. 
18. Verify Historical query analysis has all the data prior to the disaster also. 
19. Verify Audit logs have all the data prior to the disaster also. 
20. Verify completed jobs on Activity Monitor has all the data prior to the disaster.