Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace
...
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Important
|
...
AWS CloudFormation template. Contact Kyvos support to get your custom template. Alternatively, you can download the Kyvos_AWS_Default_Template_EMR2023.3.json file from the AWS Installation Files folder or create a template as per your requirements.
The CloudFormation template can be deployed through the logged-in user or a role. The logged-in user must have the required policies given in the aws-console-user-iam-policy.json file in the AWS Installation Files folder.
Both primary and disaster recovery CloudFormation deployment templates now have automated key pairs creation.
Anchor step3 step3 Storage Location: The generated key pairs are stored in the designated Amazon S3 buckets.
The key pairs are stored in the following locations:
For Primary Deployment: OutputBucketName/user/engine_work/keys/
For DR Deployment: CrossRegionDrBucketName/user/engine_work/keys/
Key Names: The naming convention for key pairs has been standardized to improve clarity and usability.
NOTE: The key pair for the bastion host is named bastionhost.pem, while the key pair for other nodes is named kyvos.pem.
Networking requirements:
Use the Network CloudFormation template to automatically create network resources (VPC, Subnet, and Security Group).
If you want to deploy your network with NAT Gateway, use the NATGateway Template (vpc_nat.json file) provided in the AWS Installation Files folder .
OR
If you want to use existing network resources, perform the following steps in your VPC.
You must create VPC Endpoints within your VPC to connect with the AWS services. Else, you must have the internet and NAT Gateway in the subnet.
List of VPC Endpoints for AWS services required by Kyvos:
AWS Service Name
Description/Purpose
VPC Endpoint Name
CloudWatch logs
Used to send bootstrap logs of the EC2 machines to CloudWatch Logs.
com.amazonaws.{AWS-REGION}.logs
EMR
Used to connect to EMR from the Kyvos BI Server for creating on-demand EMR and other EMR related activities
com.amazonaws.{AWS-REGION}.elasticmapreduce
Glue
Used to connect to Glue from the Kyvos BI Server and fetch metadata of the tables stored.
com.amazonaws.{AWS-REGION}.glue
Cloudformation
Used by Kyvos Manager at the time of deployment to validate and get details from the AWS stack in Cloudformation.
com.amazonaws.{AWS-REGION}.cloudformation
CloudWatch Event
Used to schedule events on CloudWatch Event for scheduled starting of the Kyvos BI Server.
com.amazonaws.{AWS-REGION}.events
S3
Used to connect to an S3 bucket for reading raw data and writing metadata.
com.amazonaws.{AWS-REGION}.s3
RDS
Used for scheduled start/stop of the Kyvos cluster along with RDS.
com.amazonaws.{AWS-REGION}.rds
EC2
Used by Kyvos Manager to describe EC2 and Kyvos BI Server for scheduled start/stop of Query Engines.
com.amazonaws.{AWS-REGION}.ec2
Secrets Manager
Used by the Kyvos BI Server to get the passwords stored in AWS Secrets Manager.
com.amazonaws.${AWS-REGION}.secretsmanager
...
Verify that the EMR default roles exist in your AWS account. If they do not exist, you must execute the following command:
aws emr create-default-rolesYou can create IAM roles using the CloudFormation template (automated_deployment_iam_role.json file) provided in the AWS Installation Files folder.
ORCreate IAM Role for:
Refer to the section Creating IAM Roles for EC2 and Lambda to create new roles.EC2 that will be attached to all Kyvos instances. This role contains all the permissions required by Kyvos Services and Kyvos Manager.
Details for permissions required for EC2.Lambda that will be attached to the Kyvos created Lambda functions. This role contains all the permissions required by lambda functions to run.
S3 Bucket permissions
If you want to use an existing S3 bucket and IAM role, or if you want to read data from an S3 bucket other than where Kyvos is deployed, then the IAM role must have the following permissions on the S3 bucket.
Here, replace:
<Bucket Name> with the name of your bucket name.
<Lambda Role> with the name of your Lambda Role.
<EC2 Role> with the name of your EC2 Role.
Code Block { "Version": "2008-10-17", "Statement": [ { "Sid": "Ec2LambdaRoleBucketPolicy", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::<AWS Accout ID>:role/EMR_EC2_DefaultRole", "arn:aws:iam::<AWS Accout ID>:role/<Lambda Role>", "arn:aws:iam::<AWS Accout ID>:role/<EC2 Role>" ] }, "Action": [ "s3:PutAnalyticsConfiguration", "s3:GetObjectVersionTagging", "s3:ReplicateObject", "s3:GetObjectAcl", "s3:GetBucketObjectLockConfiguration", "s3:DeleteBucketWebsite", "s3:PutLifecycleConfiguration", "s3:GetObjectVersionAcl", "s3:DeleteObject", "s3:GetBucketPolicyStatus", "s3:GetObjectRetention", "s3:GetBucketWebsite", "s3:PutReplicationConfiguration", "s3:PutObjectLegalHold", "s3:GetObjectLegalHold", "s3:GetBucketNotification", "s3:PutBucketCORS", "s3:GetReplicationConfiguration", "s3:ListMultipartUploadParts", "s3:PutObject", "s3:GetObject", "s3:PutBucketNotification", "s3:PutBucketLogging", "s3:GetAnalyticsConfiguration", "s3:PutBucketObjectLockConfiguration", "s3:GetObjectVersionForReplication", "s3:GetLifecycleConfiguration", "s3:GetInventoryConfiguration", "s3:GetBucketTagging", "s3:PutAccelerateConfiguration", "s3:DeleteObjectVersion", "s3:GetBucketLogging", "s3:ListBucketVersions", "s3:RestoreObject", "s3:ListBucket", "s3:GetAccelerateConfiguration", "s3:GetBucketPolicy", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:GetObjectVersionTorrent", "s3:AbortMultipartUpload", "s3:GetBucketRequestPayment", "s3:GetObjectTagging", "s3:GetMetricsConfiguration", "s3:DeleteBucket", "s3:PutBucketVersioning", "s3:GetBucketPublicAccessBlock", "s3:ListBucketMultipartUploads", "s3:PutMetricsConfiguration", "s3:GetBucketVersioning", "s3:GetBucketAcl", "s3:PutInventoryConfiguration", "s3:GetObjectTorrent", "s3:PutBucketWebsite", "s3:PutBucketRequestPayment", "s3:PutObjectRetention", "s3:GetBucketCORS", "s3:GetBucketLocation", "s3:ReplicateDelete", "s3:GetObjectVersion", "s3:PutBucketTagging" ], "Resource": [ "arn:aws:s3:::bucket_name/*", "arn:aws:s3:::bucket_name" ] } ] }
You must have the Access Key and Secret Key to access the Kyvos bundle. Contact Kyvos Support for details.
Valid Kyvos license file.
...