Page Comparison

Applies to: Image Removed Kyvos Enterprise   Image Removed Kyvos Cloud (Managed Services on AWS)    Image Removed Kyvos Azure Marketplace Image Removed Kyvos AWS Marketplace    Image Removed Kyvos Applies to: Kyvos Enterprise  Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)    Image Removed Kyvos Free ( Limited offering for AWS)

...

Kyvos provides the following methods for wizard-based deployment on AWS:

1. Using Kyvos public AMI

2. Using Custom AMI

Depending upon your type of installation, you will need to ensure that the corresponding prerequisites are available.

...

.

Common prerequisites

Regardless of the type of installation, the following prerequisites should be available.

...

1. EC2  key pair, consisting of a private key and a public key. You can

...

1. create

...

1. the key pair if needed.

Networking requirements

...

1. Use the

...

1. Network

...

1. CloudFormation template

...

1. to automatically create network resources (VPC, Subnet, and Security Group)

...

1. . 

   1. If you want to deploy your network with NAT Gateway,

...

1. 1. use the

...

1. 1. NATGateway Template (vpc_nat.json

...

1. 1. file in the installation folder).

   2. If you want to deploy your network with Endpoints, use the Endpoints Template ( vpc_internet_gateway.json file in the installation folder).
      OR

   3. If you want to use existing network resources, perform the following steps in your VPC. 

      1. You must create VPC Endpoints within your VPC to connect with the AWS services. Else, you must have the internet and NAT Gateway in the subnet.

         List of VPC Endpoints for AWS services required by Kyvos:

Panel
panelIconId atlassian-note panelIcon :note: bgColor #DEEBFF
In the table below, change the {AWS-REGION} according to the region in which you are deploying Kyvos. AWS does not provide a VPC endpoint for the Cost explorer service, so the Kyvos Resource Usage feature will not work without internet access.

AWS Service Name	Description/Purpose	VPC Endpoint Name
CloudWatch logs	Used to send bootstrap logs of the EC2 machines to CloudWatch Logs.	com.amazonaws.{AWS-REGION}.logs

...

EMR	Used to connect

...

to EMR from the Kyvos BI Server for

...

creating on-demand EMR and other

...

EMR related activities	com.amazonaws.{AWS-REGION}.elasticmapreduce
Glue	Used to connect to Glue from the Kyvos BI Server and fetch metadata of the tables stored.	com.amazonaws.{AWS-REGION}.glue
Cloudformation	Used by Kyvos Manager at the time of deployment to validate and get details from the AWS stack in Cloudformation.	com.amazonaws.{AWS-REGION}.cloudformation
CloudWatch Event	Used to schedule events on CloudWatch Event for scheduled starting of the Kyvos BI Server.	com.amazonaws.{AWS-REGION}.events
S3	Used to connect to S3 bucket for reading raw data and writing metadata.	com.amazonaws.{AWS-REGION}.s3
RDS	Used for scheduled start/stop of the Kyvos cluster along with RDS.	com.amazonaws.{AWS-REGION}.rds
EC2	Used by Kyvos Manager to describe EC2 and Kyvos BI Server for scheduled start/stop of Query Engines.	com.amazonaws.{AWS-REGION}.ec2
Secrets Manager	Used by the Kyvos BI Server to get the passwords stored in AWS Secrets Manager.	com.amazonaws.${AWS-REGION}.secretsmanager

In the table above, change the {AWS-REGION} according to the region in which you are deploying Kyvos.
AWS does not provide a VPC endpoint for the Cost explorer service, so the Kyvos Resource Usage feature will not work without internet access.

...

Panel
panelIconId atlassian-info panelIcon :info: bgColor #FFFAE6
Important If you are using an existing EMR in another VPC, you must enable peering between the Kyvos VPC and the EMR VPC.

Permission requirements

1. Verify that the EMR default roles exist in your AWS account. If they do not exist, you must execute the following command: 
   aws emr create-default-roles

   1. You can create IAM roles using the CloudFormation template (wizard_based_deployment_iam_role.json

...

1. 1.  file).
      OR

   2. Create

...

1. 1. IAM Role

...

1. 1. for:

      1. EC2 that will be attached to all Kyvos instances.  This role contains all the permissions required by Kyvos Services and Kyvos Manager. 
         Details for permissions required for EC2. You need to provide all the permissions mentioned in the Permissions required for Automated CloudFormation template-based deployment and Additional permissions required for Wizard-based deployment sections.

      2. Lambda that will be attached to the Kyvos-created Lambda functions.

...

1. 1. 1. This role contains all the permissions required by lambda functions to run.
         Download

...

1. 1. 1. the ec2_iam_policy.json

...

1. 1. 1. and

...

1. 1. 1. lambda_iam_policy.json

...

1. 1. 1. files in the installation folder.

2. Create Databricks-instanceprofile-role with the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GrantCatalogAccessToGlue",
            "Effect": "Allow",
            "Action": [
                "glue:BatchCreatePartition",
                "glue:BatchDeletePartition",
                "glue:BatchGetPartition",
                "glue:CreateDatabase",
                "glue:CreateTable",
                "glue:CreateUserDefinedFunction",
                "glue:DeleteDatabase",
                "glue:DeletePartition",
                "glue:DeleteTable",
                "glue:DeleteUserDefinedFunction",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetUserDefinedFunction",
                "glue:GetUserDefinedFunctions",
                "glue:UpdateDatabase",
                "glue:UpdatePartition",
                "glue:UpdateTable",
                "glue:UpdateUserDefinedFunction"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": "*"
        }
    ]
}

3. S3 Bucket permissions for using existing bucket

   If you want to use an existing S3 bucket and IAM role, or if you want to read data from an S3 bucket other than where Kyvos is deployed, then the IAM role must have the following permissions on the S3 bucket.

...

Here, replace:
<Bucket Name> with the name of your bucket name.
<Lambda Role> with the name of your Lambda Role.
<EC2 Role> with the name of your EC2 Role.

{ 
    "Version": "2008-10-17", 
    "Statement": [ 
        { 
            "Sid": "Ec2LambdaRoleBucketPolicy", 
            "Effect": "Allow", 
            "Principal": { 
                "AWS": [ 
                    "arn:aws:iam::456531263183:role/EC2-Role", 
                    "arn:aws:iam::456531263183:role/Databricks-instanceprofile-role"  
               ] 
            }, 
            "Action": [ 
                "s3:PutAnalyticsConfiguration", 
                "s3:GetObjectVersionTagging", 
                "s3:ReplicateObject", 
                "s3:GetObjectAcl", 
                "s3:GetBucketObjectLockConfiguration", 
                "s3:DeleteBucketWebsite", 
                "s3:PutLifecycleConfiguration", 
                "s3:GetObjectVersionAcl", 
                "s3:DeleteObject", 
                "s3:GetBucketPolicyStatus", 
                "s3:GetObjectRetention", 
                "s3:GetBucketWebsite", 
                "s3:PutReplicationConfiguration", 
                "s3:PutObjectLegalHold", 
                "s3:GetObjectLegalHold", 
                "s3:GetBucketNotification", 
                "s3:PutBucketCORS", 
                "s3:GetReplicationConfiguration", 
                "s3:ListMultipartUploadParts", 
                "s3:PutObject", 
                "s3:GetObject", 
                "s3:PutBucketNotification", 
                "s3:PutBucketLogging", 
                "s3:GetAnalyticsConfiguration", 
                "s3:PutBucketObjectLockConfiguration", 
                "s3:GetObjectVersionForReplication", 
                "s3:GetLifecycleConfiguration", 
                "s3:GetInventoryConfiguration", 
                "s3:GetBucketTagging", 
                "s3:PutAccelerateConfiguration", 
                "s3:DeleteObjectVersion", 
                "s3:GetBucketLogging", 
                "s3:ListBucketVersions", 
                "s3:RestoreObject", 
                "s3:ListBucket", 
                "s3:GetAccelerateConfiguration", 
                "s3:GetBucketPolicy", 
                "s3:PutEncryptionConfiguration", 
                "s3:GetEncryptionConfiguration", 
                "s3:GetObjectVersionTorrent", 
                "s3:AbortMultipartUpload", 
                "s3:GetBucketRequestPayment", 
                "s3:GetObjectTagging", 
                "s3:GetMetricsConfiguration", 
                "s3:DeleteBucket", 
                "s3:PutBucketVersioning", 
                "s3:GetBucketPublicAccessBlock", 
                "s3:ListBucketMultipartUploads", 
                "s3:PutMetricsConfiguration", 
                "s3:GetBucketVersioning", 
                "s3:GetBucketAcl", 
                "s3:PutInventoryConfiguration", 
                "s3:GetObjectTorrent", 
                "s3:PutBucketWebsite", 
                "s3:PutBucketRequestPayment", 
                "s3:PutObjectRetention", 
                "s3:GetBucketCORS", 
                "s3:GetBucketLocation", 
                "s3:ReplicateDelete", 
                "s3:GetObjectVersion", 
                "s3:PutBucketTagging" 
            ], 
            "Resource": [ 
                "arn:aws:s3:::bucket-name/*", 
                "arn:aws:s3:::bucket-name" 
            ] 
        } 
    ] 
}

...

4. You must have the Access Key and Secret Key to access the Kyvos bundle.

...

4.  Contact Kyvos Support

...

4.  for details.

5. Valid Kyvos license file.

...

4. Databricks cluster with the following

...

4. parameters:

...

4. 1. Databricks Runtime Version:

...

4. 1. Select 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)

   2. Autopilot Options:

...

4. 1. Select the following:

...

4. 1. 1. Enable autoscaling:

...

4. 1. 1.  Select this to enable autoscaling.  

      2. Terminate after ___ minutes of inactivity. Set the value as

...

4. 1. 1. 30.  

   2. Worker type:

...

4. 1. Recommended

...

4. 1. value r5.4xlarge  

      1. Min Workers: Recommended value 1

...

4. 1. 2. Max Workers: Recommended value 10

   2. Driver Type:

...

4. 1. Recommended value r 5.xlarge  

   2. Advanced

...

4. 1. options   

...

4. 1. 1. To use Databricks with Spot Instances:

         • Select the Spot fall back to On-demand checkbox from the On-demand/spot composition area.

         • Specify the number of workers.

      2. In the

...

4. 1. 1. Spark Configurations

...

4. 1. 1. define the

...

4. 1. 1. following property

...

4. 1. 1. in case of Glue-based deployment.

...

...

4. 1. 1. • spark.databricks.hive.metastore.glueCatalog.enabled

...

4. 1. 1. • =true  

      2. If cross-account glue is to be used, then define the following property to access cross-account glue :  

         • spark.hadoop.hive.metastore.glue.catalogid <GLUE_CATALOG_ID>  

      3. After these, set the below parquet-specific configuration properties:  

         • spark.hadoop.spark.sql.parquet .int96AsTimestamp true  

         • spark.sql.parquet.binaryAsString false  

         • spark.sql.parquet .int96AsTimestamp true  

         • spark.hadoop.spark.sql.parquet.binaryAsString false

         • spark.databricks.preemption.enabled false

         • spark.sql.caseSensitive false

         • spark.hadoop.spark.sql.caseSensitive false

   2. Tags: Owner and

...

4. 1. JIRA tags are required to run the cluster.  

...

4. 1. Instance profile: Copy the Instance Profile ARN of the role created

...

4. 1. earlier (Point 2 of the Permission requirements).  

      1. In Databricks console, go to Admin Console > Instance Profile and click Add Instance Profile. Paste the Instance Profile ARN in the text box.  

      2. Select the Skip Validation checkbox and then click Add.  

      3. In Cluster settings, go

...

4. 1. 1. to Advance Options, and in Instance Profile field, select the instance profile created above.  

5. Databricks information:

...

4. 1. Databricks Cluster Id: To obtain this ID, click the Cluster Name on the Clusters page in Databricks.  
      The page URL shows <https://<databricks-instance>/#/settings/clusters/<cluster-id>. The cluster ID is the number after the

...

4. 1. /

...

4. 1. cluster/

...

4. 1. component in the URL of this page.

...

4. 2. Databricks Cluster Organization ID:

...

4. 1.  To obtain this ID, click the Cluster Name on the Clusters page in Databricks.  
      The number after o= in the workspace URL is the organization ID. For example, if the workspace URL is https://westus.azuredatabricks.net/?o=7692xxxxxxxx, then the organization ID is 7692xxxxxxxx.

...

4. 2. Databricks Role ARN: Use the ARN of the Databricks-instanceprofile-role created

...

4. 1. earlier (Point 2 of the Permission requirements). 
      The ARN looks like this: arn:aws:iam ::45653****** *:role /AssumeRoleTest

...

4. 1. 
      This Databricks Role should have " iam:PassRole" permission in the role you have created for the

...

4. 1. Databricks workspace.  

5. If using an existing Secrets Manager, ensure that the KYVOS-CONNECTION-DATABRICKS-TOKEN key is added to it.

Using Kyvos Public AMI

In addition to the prerequisites mentioned in the Common section, you must have the following:

1. AWS CloudFormation template to create the resources stack. You can download the default template ( Kyvos_Template_AWS_DATABRICKS.json) provided in the Installation Files folder or create

...

1. a template as per your requirements.

2. The CloudFormation template can be deployed through the logged-in user or a role. The logged-in user must have the required policies given in the aws-console-user-iam-policy.json file.

Using Custom AMI

In addition to the prerequisites mentioned in the Common section, you must have the following:

1. AWS-CLI should be installed on all Kyvos Instances. To install perform the following steps.

   1. Install zip to unzip AWS CLI setup.

      Code Block
      yum install unzip

   2. Execute the following commands:

      Code Block
      curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install

   3. Open /etc/bashrc in any of the command line editors and add the below text in the file:

      Code Block
      export PATH=/usr/local/bin/:$PATH source /etc/bashrc

2. Increase ulimit of Kyvos user on all nodes using the command:

   Code Block
   echo “kyvos hard nofile 10240” >> /etc/security/limits.conf  echo “kyvos hard nofile 10240” >> /etc/security/limits.d/20-nproc.conf echo “kyvos soft nofile 10240” >> /etc/security/limits.conf  echo “kyvos soft nofile 10240” >> /etc/security/limits.d/20-nproc.conf

3. Install open-ssl on Kyvos Manager node to enable TLS using the command:

   Code Block
   yum install openssl

4. Download and copy the file to /sbin/

   Code Block
   cd /sbin/ && wget https://expanse.kyvosinsights.com/s/dKEtQeQLnszNwL6/download -O ebsnvme-id  chmod a+x /sbin/ebsnvme-id

5. Create below directories on all nodes with 750 permissions and make Kyvos user owner of these directories.

   Code Block
   /data/kyvos/app /data/kyvos/installs

6. Create the directories and assign 777 permissions on them:

   Code Block
   sudo mkdir -p /mnt/s3  sudo mkdir /mnt/tmp chmod -R 777 /mnt

7. Ensure that the required ports are available.

8. Ensure that the required OS Commands used by Kyvos Manager are available on all the machines.