Prerequisites for Wizard-based Deployment on AWS with EMR

Applies to: Kyvos Enterprise  Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Kyvos provides the following methods for wizard-based deployment on AWS:

1. Using Kyvos public AMI

2. Using Custom AMI

Depending upon your type of installation, you will need to ensure that the corresponding prerequisites are available.

Common prerequisites

Regardless of the type of installation, the following prerequisites should be available.

1. EC2  key pair, consisting of a private key and a public key. You can create the key pair if needed.

Networking requirements

1. Use the Network CloudFormation template to automatically create network resources (VPC, Subnet, and Security Group). 

   1. If you want to deploy your network with NAT Gateway, use the NATGateway Template (vpc_nat.json file in the installation folder).

   2. If you want to deploy your network with Endpoints, use the Endpoints Template ( vpc_internet_gateway.json file in the installation folder).
      OR

   3. If you want to use existing network resources, perform the following steps in your VPC. 

      1. You must create VPC Endpoints within your VPC to connect with the AWS services. Else, you must have the internet and NAT Gateway in the subnet.

         List of VPC Endpoints for AWS services required by Kyvos:

Permission requirements

1. Verify that the EMR default roles exist in your AWS account. If they do not exist, you must execute the following command: 
   aws emr create-default-roles

   1. You can create IAM roles using the CloudFormation template (wizard_based_deployment_iam_role.json  file).
      OR

   2. Create IAM Role for:

      1. EC2 that will be attached to all Kyvos instances.  This role contains all the permissions required by Kyvos Services and Kyvos Manager. 
         Details for permissions required for EC2. You need to provide all the permissions mentioned in the Permissions required for Automated CloudFormation template-based deployment and Additional permissions required for Wizard-based deployment sections.

      2. Lambda that will be attached to the Kyvos-created Lambda functions. This role contains all the permissions required by lambda functions to run.
         Download the ec2_iam_policy.json and lambda_iam_policy.json files in the installation folder.

2. S3 Bucket permissions for using existing bucket

   If you want to use an existing S3 bucket and IAM role, or if you want to read data from an S3 bucket other than where Kyvos is deployed, then the IAM role must have the following permissions on the S3 bucket.

Here, replace:

<Bucket Name>  with the name of your bucket name.

<Lambda Role>  with the name of your Lambda Role.

<EC2 Role>  with the name of your EC2 Role.

{​​
"Version": "2008-10-17",
"Statement": [
{​​
"Sid": "Ec2LambdaRoleBucketPolicy",
"Effect": "Allow",
"Principal": {​​
"AWS": [
"arn:aws:iam::<AWS Accout ID>:role/EMR_EC2_DefaultRole",
"arn:aws:iam::<AWS Accout ID>:role/<Lambda Role>",
"arn:aws:iam::<AWS Accout ID>:role/<EC2 Role>"
]
}​​,
"Action": [
"s3:PutAnalyticsConfiguration",
"s3:GetObjectVersionTagging",
"s3:ReplicateObject",
"s3:GetObjectAcl",
"s3:GetBucketObjectLockConfiguration",
"s3:DeleteBucketWebsite",
"s3:PutLifecycleConfiguration",
"s3:GetObjectVersionAcl",
"s3:DeleteObject",
"s3:GetBucketPolicyStatus",
"s3:GetObjectRetention",
"s3:GetBucketWebsite",
"s3:PutReplicationConfiguration",
"s3:PutObjectLegalHold",
"s3:GetObjectLegalHold",
"s3:GetBucketNotification",
"s3:PutBucketCORS",
"s3:GetReplicationConfiguration",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:GetObject",
"s3:PutBucketNotification",
"s3:PutBucketLogging",
"s3:GetAnalyticsConfiguration",
"s3:PutBucketObjectLockConfiguration",
"s3:GetObjectVersionForReplication",
"s3:GetLifecycleConfiguration",
"s3:GetInventoryConfiguration",
"s3:GetBucketTagging",
"s3:PutAccelerateConfiguration",
"s3:DeleteObjectVersion",
"s3:GetBucketLogging",
"s3:ListBucketVersions",
"s3:RestoreObject",
"s3:ListBucket",
"s3:GetAccelerateConfiguration",
"s3:GetBucketPolicy",
"s3:PutEncryptionConfiguration",
"s3:GetEncryptionConfiguration",
"s3:GetObjectVersionTorrent",
"s3:AbortMultipartUpload",
"s3:GetBucketRequestPayment",
"s3:GetObjectTagging",
"s3:GetMetricsConfiguration",
"s3:DeleteBucket",
"s3:PutBucketVersioning",
"s3:GetBucketPublicAccessBlock",
"s3:ListBucketMultipartUploads",
"s3:PutMetricsConfiguration",
"s3:GetBucketVersioning",
"s3:GetBucketAcl",
"s3:PutInventoryConfiguration",
"s3:GetObjectTorrent",
"s3:PutBucketWebsite",
"s3:PutBucketRequestPayment",
"s3:PutObjectRetention",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:ReplicateDelete",
"s3:GetObjectVersion",
"s3:PutBucketTagging"
],
"Resource": [
             "arn:aws:s3:::bucket_name/*",
             "arn:aws:s3:::bucket_name"
            ]
        }
    ]
}

3. To deploy the Kyvos cluster through the Kyvos Manager wizard method using  password-based authentication for service nodes, ensure that the permissions listed here are available on all the EC2 instances for Linux user deploying the cluster.

4. You must have the Access Key and Secret Key to access the Kyvos bundle. Contact Kyvos Support for details.

5. Valid Kyvos license file.

Using Kyvos Public AMI

In addition to the prerequisites mentioned in the Common section, you must have the following:

1. AWS CloudFormation template to create the resources stack. You can download the default template (Kyvos_Template_AWS_EMR.json file), or create a template as per your requirements.

2. The CloudFormation template can be deployed through the logged-in user or a role. The logged-in user must have the required policies given in the  aws-console-user-iam-policy.json file.

Using Custom AMI

In addition to the prerequisites mentioned in the Common section, you must have the following:

1. AWS-CLI should be installed on all Kyvos Instances. To install, perform the following steps.

   1. Install zip to unzip AWS CLI setup.

      yum install zip unzip

   2. Execute the following commands:

      curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
      unzip awscliv2.zip
      sudo ./aws/install

   3. Open /etc/bashrc in any of the command line editors and add the below text in the file:

      export PATH=/usr/local/bin/:$PATH
      source /etc/bashrc

2. Increase ulimit of Kyvos user on all nodes using the command:

   echo “kyvos hard nofile 10240” >> /etc/security/limits.conf 
   echo “kyvos hard nofile 10240” >> /etc/security/limits.d/20-nproc.conf
   echo “kyvos soft nofile 10240” >> /etc/security/limits.conf 
   echo “kyvos soft nofile 10240” >> /etc/security/limits.d/20-nproc.conf 

3. Install open-ssl on Kyvos Manager node to enable TLS using the command:

   yum install openssl

4. Install wget to download ebsnvme file.

   yum install wget 

5. Download and copy the file to /sbin/

   cd /sbin/ && wget https://expanse.kyvosinsights.com/s/dKEtQeQLnszNwL6/download -O ebsnvme-id 
   chmod a+x /sbin/ebsnvme-id

6. Create below directories on all nodes with 750 permissions and make Kyvos user owner of these directories.

   /data/kyvos/app
   /data/kyvos/installs

7. Create the directories and assign 777 permissions on them:

   sudo mkdir -p /mnt/s3 
   sudo mkdir /mnt/tmp
   chmod -R 777 /mnt  

8. Ensure that the required ports are available.

9. Ensure that the required OS Commands used by Kyvos Manager are available on all the machines.