Automated deployment for AWS via CloudFormation with No-Spark

Prerequisites

Before you start the automated installation for Kyvos on AWS, ensure you have the following information.

Before starting the deployment for AWS, you must have the following.

1. AWS CloudFormation template. Contact  Kyvos support to get your custom template. Alternatively, you can download the default K8 template file from the AWS Installation Files folder or create a template as per your requirements.

2. The CloudFormation template can be deployed through the logged-in user or a role. The logged-in user must have the required policies given in the aws-console-user-iam-policy.json file.

3. EC2 key pair, consisting of a private key and a public key. You can create the key pair if needed.

4. Networking requirements:

   1. Use the Network CloudFormation template to create network resources (VPC, Subnet, and Security Group) automatically. 

      1. If you want to deploy your network with NAT Gateway, use the NATGateway Template (vpc_nat.json file) . 
         OR

      2. If you want to use existing network resources, perform the following steps in your VPC. 

   2. Refer to Amazon documentation for VPC and Subnet requirement for EKS cluster. 

      1. Ensure that at least 8 IP addresses are available in the public subnets, each located in a different Availability Zone within the VPC where your EKS cluster is created. This is required by the load balancer, which is created by the EKS cluster's load balancer.

      2. You must create VPC Endpoints within your VPC, to connect with the AWS services. Else, you must have the internet and NAT Gateway in the subnet.

         List of VPC Endpoints for AWS services required by Kyvos:
         
         

5. Permission requirements:

   1. You can create IAM roles using the CloudFormation template (automated_deployment_iam_role.json file).
      OR

   2. Create IAM Role for:
      Refer to the section Creating IAM Roles for EC2 and Lambda to create new roles.

      1. EC2 that will be attached to all Kyvos instances. This role contains all the permissions required by Kyvos Services and Kyvos Manager.
         Details for permissions required for EC2.

      2. Lambda that will be attached to the Kyvos created Lambda functions. This role contains all the permissions required by lambda functions to run.

6. S3 Bucket permissions

   If you want to use an existing S3 bucket and IAM role, or if you want to read data from an S3 bucket other than where Kyvos is deployed, then the IAM role must have the following permissions on the S3 bucket.

   Here, replace:

   <Bucket Name> with the name of your bucket name.

   <Lambda Role> with the name of your Lambda Role.

   <EC2 Role> with the name of your EC2 Role.
   <AWS Account ID> With your AWS account ID.
   <kyvosEksOidcrole> With the OIDC role name that will be attached to the add-on of EKS cluster.
   <NodeGroupRole> With the role name that will be attached to your EKS cluster’s node group.

   { "Version": "2008-10-17", "Statement": [ { "Sid": "Ec2LambdaRoleBucketPolicy", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::<AWS Accout ID>:role/EC2-Role", "arn:aws:iam::<AWS Accout ID>:role/kyvosEksOidcrole", "arn:aws:iam::<AWS Accout ID>:role/NodeGroupRole" "arn:aws:iam::<AWS Accout ID>:role/<Lambda Role>", ] }, "Action": [ "s3:PutAnalyticsConfiguration", "s3:GetObjectVersionTagging", "s3:ReplicateObject", "s3:GetObjectAcl", "s3:GetBucketObjectLockConfiguration", "s3:DeleteBucketWebsite", "s3:PutLifecycleConfiguration", "s3:GetObjectVersionAcl", "s3:DeleteObject", "s3:GetBucketPolicyStatus", "s3:GetObjectRetention", "s3:GetBucketWebsite", "s3:PutReplicationConfiguration", "s3:PutObjectLegalHold", "s3:GetObjectLegalHold", "s3:GetBucketNotification", "s3:PutBucketCORS", "s3:GetReplicationConfiguration", "s3:ListMultipartUploadParts", "s3:PutObject", "s3:GetObject", "s3:PutBucketNotification", "s3:PutBucketLogging", "s3:GetAnalyticsConfiguration", "s3:PutBucketObjectLockConfiguration", "s3:GetObjectVersionForReplication", "s3:GetLifecycleConfiguration", "s3:GetInventoryConfiguration", "s3:GetBucketTagging", "s3:PutAccelerateConfiguration", "s3:DeleteObjectVersion", "s3:GetBucketLogging", "s3:ListBucketVersions", "s3:RestoreObject", "s3:ListBucket", "s3:GetAccelerateConfiguration", "s3:GetBucketPolicy", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:GetObjectVersionTorrent", "s3:AbortMultipartUpload", "s3:GetBucketRequestPayment", "s3:GetObjectTagging", "s3:GetMetricsConfiguration", "s3:DeleteBucket", "s3:PutBucketVersioning", "s3:GetBucketPublicAccessBlock", "s3:ListBucketMultipartUploads", "s3:PutMetricsConfiguration", "s3:GetBucketVersioning", "s3:GetBucketAcl", "s3:PutInventoryConfiguration", "s3:GetObjectTorrent", "s3:PutBucketWebsite", "s3:PutBucketRequestPayment", "s3:PutObjectRetention", "s3:GetBucketCORS", "s3:GetBucketLocation", "s3:ReplicateDelete", "s3:GetObjectVersion", "s3:PutBucketTagging" ], "Resource": [ "arn:aws:s3:::bucket-name/*", "arn:aws:s3:::bucket-name" ] } ] }

7. You must have the Access Key and Secret Key to access the Kyvos bundle. Contact Kyvos Support for details.

8. Valid Kyvos license file.

Creating CloudFormation template

The Kyvos CloudFormation template can create the following resources:

• EKS for processing semantic model and aggregations. 

• EC2 instances for Kyvos services - BI Server, Query Engines, Kyvos Manager, Kyvos Web Portal, and Postgres.

• S3 for storing Kyvos semantic model

• RDS for use as Kyvos repository if you don't want to use the default Postgres database provided in the Kyvos package.

• Lambda to use the scheduling (cluster ON) features.

• API Gateway to get the Rest URL on the Lambda function.

• CloudWatch event for scheduling the Kyvos BI Server.

• Secrets Manager for storing passwords, like Kyvos DB password, Active Directory password, and SMTP password (if configured)

• SecurityGroup for Databricks and Kyvos instances.