Deploying Kyvos (with EMR) using CloudFormation Template
Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace
Kyvos Azure Marketplace  Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)
To deploy the Kyvos using the default CloudFormation template, perform the following steps.
Log in to your AWS Console, with the user having sufficient rights as mentioned in the prerequisites.
Click the Create Stack > With New Resources (Standard)  option.
Warning
Please provide the information carefully, as you cannot edit the stack after creation.
Step 1: Specify template
In the Prerequisite - Prepare template, select the Template is ready option.
In the Specify template, select the Upload a template file option to upload your CloudFormation template using the Choose File button.
Click NEXT.
Step 2: Specify stack details
Enter a Stack name. The Stack name can contain letters (A-Z and a-z), numbers (0-9), and dashes (-).
The Kyvos cluster will be deployed in this name, and you cannot edit the name after deployment.Â
If you have enabled Web Portal High Availability, the stack name should not exceed 32 characters
The Parameters area displays the configurations defined in your template. It also allows you to enter custom values to create or update a stack.
Enter details as:
Configuration | Parameter | Description/Remarks |
---|---|---|
Security Configurations | Select the name of the Key Pair to be used with EC2 instances | A key pair consisting of a public key (stored by AWS), and a private key file (stored by the user). Together, they allow you to connect to your instance securely. |
Enter the SSH Private Key text | Provide the text of your SSH key. To get this, open your .pem file in edit mode, and copy the complete text. | |
Enable SSH for EMR/Databricks cluster | Set the value as true to enable SSH for the EMR cluster. NOTE: Currently, the only value for Databricks is false, as SSH is not supported on it. | |
S3 Configurations | Enter name of the S3 bucket for storing Kyvos semantic model | Enter the name of your existing bucket, if you selected False above. If you select True, Kyvos will create a bucket with this name. Ensure that the bucket is not already in use. NOTE: See the AWS rules for naming buckets. Additionally, Kyvos does not allow . (dot) to be used for Bucket Name. The cluster may fail to deploy if you do not meet these requirements. |
Network Configurations | VPC | Select the VPC in which EC2 instances will be launched. NOTE:Â If you have created your VPC using the NAT Gateway template or the Internet Gateway template, select that VPC here. |
Subnet | Select the Subnet to be attached to EC2 instances. NOTE: In the case of Kyvos Web Portal HA (High Availability) or an RDS repository, you must select at least two subnets from different Availability Zones. Otherwise, you have to select only one subnet. | |
Availability Zone | Select the Availability Zone for the selected Subnet. | |
IAM Roles Configurations | Enter the name of the IAM Role that will be attached to all the Kyvos EC2 instances   | Provide the name of the IAM Role that you want to attach to the EC2 instance. Refer to the section Creating IAM Roles for EC2 and Lambda to create new roles. |
Enter the name of the IAM Role that will be attached to all the Lambda functions   | Provide the name of the IAM Role that you want to attach to the Lambda function. Refer to the section Creating IAM Roles for EC2 and Lambda to create new roles. | |
Custom Prefix Configurations | Custom Prefix Ec2 Instances | Enter the prefix you want to append to the names of EC2 instances used with Kyvos.  NOTE: The prefix can be up to 20 characters long and must begin and end with a word character. |
Custom Prefix Volumes | Enter the prefix you want to append to the names of Volumes used with Kyvos.  NOTE: The prefix can be up to 20 characters long and must begin and end with a word character. | |
Custom Prefix Other Services | Enter the prefix to be append before the services. These services include Security Groups, EMR , Load balancer, and RDS .  NOTE: The prefix can be up to 20 characters long and must begin and end with a word character. | |
Custom Prefix Secret Manager | Enter the prefix you want to append to the Secrets Manager used with Kyvos. NOTE: The prefix can be up to 20 characters long and must begin and end with a word character. The parameter is displayed only when you select the Deploy with Enhanced Security checkbox while creating the CFT template. | |
Load Balancer Configuration | PublicSubnets   | Select 2 public subnets, each from a different availability zone. Make sure that public subnet's availability zone matches the availability zone of the private subnets in which Kyvos instances are launched. |
RDS Configurations | Create a new subnet group for RDS? | Select True to create a new subnet group for RDS. |
Enter the name of existing Subnet group | If creating a new subnet group for RDS is false, enter the name of the existing subnet group. | |
Specify whether the database instance is a multiple Availability Zone deployment | Select True to specify the database instance is a multiple availability zone deployment.  | |
Choose the version of RDS which needs to be launched | Enter the RDS version to be launched. NOTE: From Kyvos 2023.2 onwards, RDS Postgres version 13.11 is supported for both new deployments and upgrades. Support for version 13.6 is discontinued. | |
EMR Configurations | Enter the number of Core EC2 Instances to be launched with EMRÂ | The number of Core EC2 Instances to be launched with EMR. Recommended value: 2 |
Enter the minimum number of Core EC2 instances that should be kept running | The minimum number of core nodes that will keep running during scale-in operation. Recommended value: 2 | |
Enter the maximum number of Core EC2 instances that should be kept running | The maximum number of core nodes that can be used by the EMR during scale-out operation. Recommended value: 10 | |
Enter the version of EMR which needs to be launched | Select the EMR version to be used for the Kyvos cluster.  The allowed value is EMR- 6 . 10 .0. NOTE: EMR-6.10.0 is supported only if you enable Livy on the cluster.  You will see the EMR version followed by CERTIFIED indicating if the EMR version is supported by Kyvos. | |
Enable In-transit encryption for EMR cluster | Set the value of this parameter to true if you need in-transit encryption with TLS for the EMR cluster. | |
Enter the S3Object ARN where PEM based certificate is located | Enter the S3Object Amazon Resource Name (ARN) of the KMS certificate. NOTE: This is required only if you have enabled in-transit encryption for the EMR cluster. | |
Kyvos Configurations | Enter the Access key | Access key to access the Kyvos bundle. Contact us if you haven't received it yet. |
Enter the Secret key | Secret key to access the Kyvos bundle. Contact us if you haven't received it yet. | |
Kyvos Work Directory | Enter the work directory path to be used by Kyvos. | |
Kyvos License Information | Enter the temporary Kyvos license key provided in your onboarding email. Copy the content of the license file here. | |
Enable Environment validation | Select true to validate the environment information that you provided for where the Kyvos cluster needs to be deployed. If you select false, the deployment will continue without validation. | |
Show hostname for cluster deployment | Select true to use the hostname for cluster deployment. | |
DatabaseKmRepo | Enter the name of database to be used for Kyvos Manager Repository. NOTE: This parameter is displayed only when you select the Create New RDS option during the Kyvos template creation. | |
KyvosManagerRepoDBInstanceIdentifierName | Instance Name of shared RDS NOTE: This parameter is displayed only when you select the Existing RDS option during the Kyvos template creation. | |
RDSPasswordKmRepo | Specifies the value of the password used for Kyvos Manager Postgres NOTE: This parameter is displayed only when you specify the KyvosManagerRepoDBInstanceIdentifierName parameter. | |
PostgresPassword | Provide the password used for Postgres. | |
RDSPasswordKmRepo | Specifies the value of the password used for KyvosManager Repository. The password can include any printable ASCII character except "/", """, "@" and single quote. NOTE: This parameter is displayed only when you select the Create New RDS option during the Kyvos template creation. | |
Kyvos Credit Load Information | Enter a valid Kyvos credit load key. | |
Kyvos Shared Credit Information Existing Repository | Enter the endpoint and port of the Shared Credit Information Repository.  NOTE: You MUST add a semicolon after providing the endpoint and specify the port. For example, endpoint: port. | |
Kyvos Credit Information Existing Secret | Enter the secret name used for Shared Credit Information Repository. | |
SecretName | Enter the name of your existing Secret Manager. If blank, a new Secret Manager will be created automatically. | |
Allowed IP Range  | Provide the range of IP addresses allowed to access Kyvos Instances. Use 0.0.0.0/0 to allow all users access.  | |
UsernameKmRepo | Enter the username to be used for connecting to the Kyvos Manager Repository. NOTE: This parameter is displayed only when you select the Create New RDS option during the Kyvos template creation. | |
Enable Livy Server for EMR cluster | If you have selected the EMR version as 6. 10 .0, you MUST enable Livy . Â | |
PublicSubnets | For deployments with Web portal high availability, if you selected a public subnet while creating the template, please select two public subnets. | |
AMI Configurations  (Displayed only if you selected the Custom Image option at the time of creating the template) | Enter the AMI default logged in Linux user | Each Linux instance launches with a default Linux system user account. For more details, refer to AWS documentation. |
Enter the AMI ID to be used for launching Kyvos Instances |  Provide the AMI ID. |
Click NEXT.
Step 3: Configure stack options
Tags:Â You can specify tags (key-value pairs) to apply to resources in your stack. You can add up to 50 unique tags for each stack.
Note
If deploying in a secure environment with Ranger deployed on AWS, then you MUST provide the tag as UsedBy=Kyvos
DO NOT use the UsedBy tag for any other deployments.
Permissions:Â Choose an IAM role to explicitly define how CloudFormation can create, modify, or delete resources in the stack. If you don't choose a role, CloudFormation uses permissions based on your user credentials.
Advanced options: Optionally, you can set additional options for your stack, like notification options and a stack policy.Â
Enter details as:
Parameter | Description/Remarks |
---|---|
Stack policy | Defines the resources that you want to protect from unintentional updates during a stack update. Kyvos does not allow any updates in the stack, so you can leave this blank. |
Rollback configuration | Specify alarms for CloudFormation to monitor when creating and updating the stack. If the operation breaches an alarm threshold, CloudFormation rolls it back. |
Monitoring time (optional): The number of minutes after the operation completes that CloudFormation should continue monitoring the specified alarms. | |
CloudWatch alarm (optional): Amazon Resource Name (ARN) of the alarm to monitor. | |
Notification options | Specify SNS topic ARN (optional). |
Stack creation options | Rollback on failure:Â Specifies whether the stack should be rolled back if stack creation fails. |
Timeout:Â The number of minutes before a stack creation times out. | |
Termination protection:Â Prevents the stack from being accidentally deleted. Once created, you can update this through stack actions. |
Step 4: Review
Review the settings selected so far. Click Previous if you need to change any configurations.
Step 5: Create Stack
To proceed, select the I acknowledge checkbox and click Create Stack to create and deploy the Kyvos cluster on your AWS environment.
Stack creation starts and takes around 20-25 minutes. Once the cluster is deployed, you will receive the KMAccessUrl in the Outputs tab, as shown in the following figure.
NEXT: Accessing Kyvos and Kyvos Manager
Copyright Kyvos, Inc. All rights reserved.