Creating Lambda to enable scaling of external EMR

Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Kyvos allows you to use external EMR for ROLAP queries. You can also configure scaling for this external EMR.

For this, you need to create a Lambda role. This section provides the steps and procedure for creating the Lambda role.

Note

The scaling configuration for external EMR is supported for Kyvos versions 2021.1.1 and above.

To deploy the Kyvos using the default CloudFormation template, perform the following steps.

Log in to your AWS Console, with the user having sufficient rights as mentioned in the prerequisites.
From the top-left of the screen, click the Create Stack > With New Resources (Standard) option.

Caution

Please provide the information carefully, as you cannot edit the stack after creation.

In the Prerequisite - Prepare template, select the Template is ready option.
In the Specify template, select the Upload a template file option to upload your CloudFormation template using the Choose File button.
Click NEXT.

Enter a Stack name. Stack name can contain letters (A-Z and a-z), numbers (0-9), and dashes (-). The Kyvos cluster will be deployed in this name, and you cannot edit the name after deployment.
The Parameters area displays the configurations defined in your template. It also allows you to enter custom values to create or update a stack.
Enter details as explained in the following table, and then click NEXT.

Configuration	Parameter	Description

Configuration	Parameter	Description
EMR Detail	Enter the Cluster ID of the EMR node	Provide the cluster-ID of your external EMR node.
EMR Scale-Out Schedule	Enter the number of core nodes that you want	The number of core nodes that will keep running during the scale-out event.
	Enter the time in hours when the EMR cluster should be scaled-up	Enter the time in UTC 24 hour format.
	Enter the time in minutes when the EMR cluster should be scaled-up	Enter the time in UTC 24 hour format.
EMR Scale-In Schedule	Enter the number of core nodes that you want	The number of core nodes that will keep running during the scale-in event.
	Enter the time in hours when the EMR cluster should be scaled-down	Enter the time in UTC 24 hour format.
	Enter the time in minutes when the EMR cluster should be scaled-down	Enter the time in UTC 24 hour format.
Other parameters	LambdaIamRole	To use an existing Lambda role, provide the name here. If blank, the role will be created automatically.

Tags: You can specify tags (key-value pairs) to apply to resources in your stack. You can add up to 50 unique tags for each stack.

Caution

If deploying in a secure environment with Ranger deployed on AWS, then you MUST provide the tag as UsedBy=Kyvos

DO NOT use the UsedBy tag for any other deployments.

Permissions: Choose an IAM role to explicitly define how CloudFormation can create, modify, or delete resources in the stack. If you don't choose a role, CloudFormation uses permissions based on your user credentials.
Advanced options: Optionally, you can set additional options for your stack, like notification options and a stack policy.
Enter details as:

Parameter	Description/Remarks

Parameter	Description/Remarks
Stack policy	Defines the resources that you want to protect from unintentional updates during a stack update. Kyvos does not allow any updates in the stack, so you can leave this blank.
Rollback configuration	Specify alarms for CloudFormation to monitor when creating and updating the stack. If the operation breaches an alarm threshold, CloudFormation rolls it back. Monitoring time (optional): The number of minutes after the operation completes that CloudFormation should continue monitoring the specified alarms. CloudWatch alarm (optional): Amazon Resource Name (ARN) of the alarm to monitor.
Notification options	Specify SNS topic ARN (optional).
Stack creation options	Rollback on failure: Specifies whether the stack should be rolled back if stack creation fails. Timeout: The number of minutes before a stack creation times out. Termination protection: Prevents the stack from being accidentally deleted. Once created, you can update this through stack actions.

Review the settings selected so far. Click Previous if you need to change any configurations.

To proceed, select the I acknowledge checkbox and click Create Stack to create and deploy the Kyvos cluster on your AWS environment.

Once the stack is created, you will see the status in the info as shown in the picture below.