Cluster installation using resources created for AWS with Databricks

Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Important

Before starting with the Kyvos installation, ensure to add the Security group created in Kyvos Stack to the existing Kyvos Manager instance.
Click here to see the steps for adding an existing security group to an instance.

After creating the Kyvos Manager node, you must perform the following steps.

SSH to the Kyvos Manager node using the following command:
^{ssh -i pem-file kyvos@${IP}}
Execute the following commands:
^{cd /data/kyvos/installs/}
^{aws s3 cp s3://us-east-1.kyvos/2023.1/latest/prereq/build_lib_conf.tar.gz}
^{tar -xvf build_lib_conf.tar.gz}
^cd
^{vi .bashrc}
^{export CONF_LIB_INCLUSION_HOME=/data/kyvos/installs/build_lib_conf/}
^{source .bashrc}

Access the installation URL, such as:
http://clusterIP:port/kyvosmanager/kyvos-installer/aws
If you do not have the URL and/or credentials, click here to contact support.
On the Kyvos Installer, select the I have all the resources option.
Click INSTALL KYVOS. The Setup Kyvos Environment dialog is displayed.
Enter details as:

Parameter	Description
Cluster Name	Provide a unique cluster name.
License	Upload your Kyvos license file.
Installation Path	Provide the installation path for Kyvos.
Username	Enter your username.
Authentication Type	Password: Provide your credentials. Ensure that the permissions listed here are available on all the EC2 instances for Linux user deploying the cluster. Private Key: Choose and Upload the Private Key file.
Configure Kyvos Services on Cloud	Select Instance By: IP Address: Select this option for IP Address-based deployment. Hostname: Select this option for hostname-based deployment. In this case, you can deploy the cluster using custom hostnames for nodes to be used for Kyvos services. Click the Use custom hostname for selected nodes link to specify or map a custom hostname for the selected BI Server, Query Engine, and Kyvos Manager/repository node. NOTE: Ensure that the instances have been configured for using Custom hostnames. Stack: Select the stack that you created. NOTE: If you want to deploy Kyvos Services (BI Server, Query Engine, and Web Portal) on the same Instance, in this case, select the None (node mapping Manually) option. Fetch resources: Click to view and modify any resources. The resources configured in the stack, such as EC2 instance(s) for BI Server, EC2 instance(s) for Query Engine, EC2 instance(s) for Web Portal, and so on are displayed. Click the Modify link corresponding to the Identified Resources to edit them. You can also select the Repository here. NOTE: If you want to use your own resources, select the None option, and provide the details in the displayed fields. Here, if you have the requisite permissions, your AWS resources will be displayed in the lists. Otherwise, you will have to manually provide the details. Deploy BI Server, Query Engine, and Web Portal services on ip-<>.ec2 internal instance: This checkbox is displayed only when you select the None (node mapping Manually) option from the Stack list. Selecting this checkbox will deploy Kyvos services on the same instance. NOTE: By default, the option to deploy EC2 instance(s) for BI Server, EC2 instance(s) for Query Engine, and EC2 instance(s) for Web Portal will be disabled.
Secrets Manager	Displays the name of the Secrets Manager that will be used for storing passwords. NOTE: This field is displayed if you selected the Deploy with Enhanced Security option at the time of creating resources.
Compute Cluster	Select the Databricks option.
Databricks Cluster Id	Enter the Cluster ID of Databricks Cluster.
File System Type	Select the S3 option.
Databricks Organization id	Enter the Organization ID of Databricks Workspace.
Databricks URL	Enter the URL of Databricks Workspace.

Click the Advanced Settings link. The Advanced Settings dialog box is displayed with values populated according to the ones that you defined while creating the stack.
To update any fields, enter information as:

Configuration Type	Parameter/Field	Comments/Description
Common	Kyvos Setup Package	Select the Kyvos bundle to be installed. Kyvos Manager contains compatible Kyvos bundles. To use any other Kyvos version, click the Upload button
	Cube Local Path	Enter the cube local path where cuboids will be copied. For multiple paths, enter a semicolon-separated list. NOTE: The deployment user must have full permission on the parent of the local cube path.
	Automatically restart Kyvos services when down	Select this option to enable high availability mode for the Kyvos service. This will ensure Kyvos services are always up and running. The Kyvos Manager will automatically restart the BI server, query engine, repository, and web portal services, in case any of these services go down
BI Server	Listener Port	Enter the listener port for BI Server.
	JMX Port	Enter the JMX port for BI Server health monitoring.
	Work Directory	Enter the directory location on HDFS, where BI Server has access permissions.
	Maximum Heap Memory	Enter the maximum memory available for BI Server.
	Additional Java Options	Use this to define custom Java properties for your BI Server.
Query Engine	Thrift Listener Port	Enter the thrift listener port for Query Engine.
	JMX Port	Enter the port for Query Engine health monitoring.
	Maximum Heap Memory	Enter the maximum memory for Query Engine.
	Additional Java Options	Use this to define custom Java properties for your Query Engine.
Web Portal	UI Port	Enter the port on which Kyvos Web client will run.
	JMX Port	Enter the port for Web portal health monitoring.
	Shut-down Port	Enter the port to be used for shutting the Tomcat server.
	AJP Port	Enter the port on which the AJP connector creates a socket and awaits an incoming connection.
	Maximum Heap Memory	Enter the maximum memory for the Web portal.
	Additional Java Options	Use this to define custom Java properties for the Web portal.
Network Protocol Configuration	Enable TLS	Select the checkbox to enable TLS for the cluster. Note: Before enabling TLS, if you are using your own script for resource creation instead of the Kyvos shared scripts, you must rename the OpenSSL configuration file on the Kyvos Manager and the node where the Postgres Service is running. The OpenSSL configuration file is available in the /etc/pki/tls/openssl.cnf folder. To accomplish this, you will need to have sudo access.
	HTTP Protocol Version	Select the HTTP Protocol Version for use with TLS.
	Kyvos Web Portal on HTTPS	Select the checkbox to use HTTPS for connecting to Kyvos Web Portal.
	Kyvos Web Portal Configuration	NOTE: This is displayed only if Kyvos Web Portal on HTTPS checkbox is selected Certification Mode: Select the Use Path or Upload File option. Keystore: Provide the path or upload the Keystore file for the certificate. Keystore Private Key: Provide the Keystore Private Key for the certificate. Custom Attributes for Connectors: If needed, provide attributes to be used for the HTTPS connector.

Click Apply.
Click NEXT. The system validates your settings, and all the configurations that are done so far are displayed in the Review and Confirm dialog box.
Review the settings, and click BACK to make any changes.
Select the checkboxes to accept the terms and start Kyvos services on installation.
Click INSTALL to complete the installation.

If you have installed using your custom stack or if you have used your AWS account resources in the wizard above, you can create and configure the Lamda functions to allow scheduling on the BI Server.