Versions Compared

Old Version 2

changes.mady.by.user Sarabjeet Kaur

Saved on

compared with

New Version 3

changes.mady.by.user NIDHI BHANDARI

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Applies to: (tick)Kyvos Enterprise  (error) Kyvos Cloud (SaaS on AWS) (error) Kyvos AWS Marketplace

...

Panel
panelIconIdatlassian-info
panelIcon:info:
bgColor#FFFAE6

Important

Download the files provided in the AWS Installation Files  folder folder and keep them handy.all the requisite files handy during installation and deployment. 

Common prerequisites

Regardless of the type of installation, the following prerequisites should be available.

  1. EC2 key pair, consisting of a private key and a public key. You can create the key pair if needed.

...

AWS Service Name

Description/Purpose

VPC Endpoint Name

CloudWatch logs

Used to send bootstrap logs of the EC2 machines to CloudWatch Logs.

com.amazonaws.{AWS-REGION}.logs

EMR

Used to connect to EMR from the Kyvos BI Server for creating on-demand EMR and other EMR related activities

com.amazonaws.{AWS-REGION}.elasticmapreduce

Glue

Used to connect to Glue from the Kyvos BI Server and fetch metadata of the tables stored.

com.amazonaws.{AWS-REGION}.glue

CloudformationCloudFormation

Used by Kyvos Manager at the time of deployment to validate and get details from the AWS stack in CloudformationCloudFormation.

com.amazonaws.{AWS-REGION}.cloudformation

CloudWatch Event

Used to schedule events on CloudWatch Event for scheduled starting of the Kyvos BI Server.

com.amazonaws.{AWS-REGION}.events

S3

Used to connect to S3 bucket for reading raw data and writing metadata.

com.amazonaws.{AWS-REGION}.s3

RDS

Used for scheduled start/stop of the Kyvos cluster along with RDS.

com.amazonaws.{AWS-REGION}.rds

EC2

Used by Kyvos Manager to describe EC2 and Kyvos BI Server for scheduled start/stop of Query Engines.

com.amazonaws.{AWS-REGION}.ec2

Secrets Manager

Used by the Kyvos BI Server to get the passwords stored in AWS Secrets Manager.

com.amazonaws.${AWS-REGION}.secretsmanager

...

  1. You must have the Access Key and Secret Key to access the Kyvos bundle. Contact Kyvos Support for details.

  2. Valid Kyvos license file.

  3. Databricks cluster with the following parameters:

    1. Databricks Runtime Version: Select 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)

    2. Autopilot Options: Select the following:

      1. Enable autoscaling: Select this to enable autoscaling.  

      2. Terminate after ___ minutes of inactivity. Set the value as 30.  

    3. Worker type: Recommended value r5.4xlarge  

      1. Min Workers: Recommended value 1

      2. Max Workers: Recommended value 10

    4. Driver Type: Recommended value r 5.xlarge  

    5. Advanced options   

      1. By default, the Spot fall back to On-demand checkbox is selected. Kyvos recommends you clear this checkbox.

      2. In the Spark Configurations define the following property in case of Glue-based deployment.

        • spark.databricks.hive.metastore.glueCatalog.enabled=true  

      3. If cross-account glue is to be used, then define the following property to access cross-account glue: spark.hadoop.hive.metastore.glue.catalogid <GLUE_CATALOG_ID>  

      4. After these, set the below parquet-specific configuration properties:  

        • spark.hadoop.spark.sql.parquet .int96AsTimestamp true  

        • spark.sql.parquet.binaryAsString false  

        • spark.sql.parquet .int96AsTimestamp true  

        • spark.hadoop.spark.sql.parquet.binaryAsString false

        • spark.databricks.preemption.enabled false

        • spark.sql.caseSensitive false

        • spark.hadoop.spark.sql.caseSensitive false

      5. You must change Spark configurations to use managed disk. Ensure that you must not change the configuration in the default root (/tmp) volume.

        1. In the Spark Configurations, add the spark.local.dir /local_disk0 property where the local_disk0 is the managed disk.

        2. Optionally, you can execute the df -h command from a notebook for verification.

        3. Add the SPARK_WORKER_DIR=/local_disk0 value in the Environment variables.

    6. Tags: Owner and JIRA tags are required to run the cluster.  

    7. Instance profile: Copy the Instance Profile ARN of the role created earlier (Point 2 of the Permission requirements).  

      1. In Databricks console, go to Admin Console > Instance Profile and click Add Instance Profile. Paste the Instance Profile ARN in the text box.  

      2. Select the Skip Validation checkbox and then click Add.  

      3. In Cluster settings, go to Advance Options, and in Instance Profile field, select the instance profile created above.  

  4. Databricks information:

    1. Databricks Cluster Id: To obtain this ID, click the Cluster Name on the Clusters page in Databricks.  
      The page URL shows <https://<databricks-instance>/#/settings/clusters/<cluster-id>. The cluster ID is the number after the /cluster/ component in the URL of this page.

    2. Databricks Cluster Organization ID: To obtain this ID, click the Cluster Name on the Clusters page in Databricks.  
      The number after o= in the workspace URL is the organization ID. For example, if the workspace URL is https://westus.azuredatabricks.net/?o=7692xxxxxxxx, then the organization ID is 7692xxxxxxxx.

    3. Databricks Role ARN: Use the ARN of the Databricks-instanceprofile-role created earlier (Point 2 of the Permission requirements). 
      The ARN looks like this: arn:aws:iam ::45653****** *:role /AssumeRoleTest
      This Databricks Role should have " iam:PassRole" permission in the role you have created for the Databricks workspace.  

  5. If using an existing Secrets Manager, ensure that the KYVOS-CONNECTION-DATABRICKS-TOKEN key is added to it.

Using Kyvos Public AMI

In addition to the prerequisites mentioned in the Common section, you must have the following:

...