AWSJTdCJTIydGl0bGUlMjIlM0ElMjJBV1MlMjIlN0Q=

Creating Databricks connection

To create a Databricks connection for AWS, perform the following steps.

From the Toolbox, click Setup, then choose Connections.
From the Actions menu ( ⋮ ) click Add Connection.
Enter a Name for the connection and provide information as:

Parameter	Description/Remarks
Category	Select the Process option.
Providers	Select the Databricks option.
Databricks Service Address	Enter the URL of the Databricks workspace.
Databricks Personal Access Token	Provide the personal access token to access and connect to your Databricks workspace. Refer to the Databricks documentation to know how to get your token.
Databricks Cluster Id	Enter the ID of your Databricks cluster. To obtain this ID, click the Cluster Name on the Clusters page in Databricks. The page URL shows <https://<databricks-instance>/#/settings/clusters/<cluster-id>
Is Data Process	By default, the checkbox is selected.
Use as source	Select the checkbox to use this as a read connection. In this case, the connection will be used to read data (creating registered datasets) on which the semantic model will be created.
Metastore Type	Metastore type to be used for fetching databases and table listing or writing SQL queries to design register datasets. NOTE: This option is only displayed if you select the Use as a source checkbox. You can select from the DEFAULT or GLUE METASTORE options. The GLUE option is displayed and set by default only if your AWS cluster was deployed with Glue enabled.
Hive Server JDBC URL	Databricks cluster JDBC URL to connect with Databricks internal metastore by creating JDBC connection.
SQL Engine	Select the HIVE or Spark option from the SQL Engine from the list. Is default SQL engine: To enable the connection for raw data querying, select the checkbox to set this connection to run the default SQL engine. Configure Job cluster: Use this option to allow Kyvos to execute Spark jobs on Job Cluster to reduce the costs of the process jobs. This feature is helpful and recommended in limited scenarios.
Configure Job Cluster	Use this option to allow Kyvos to execute Spark jobs on Job Cluster to reduce the costs of the process jobs. This feature is helpful and recommended in limited scenarios. Please see the recommendations and best practices sections below for details.
Autoscaling	If needed, enable autoscaling and specify the minimum and maximum number of worker nodes for the cluster. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.
Use same instance pool for worker and driver	Select the checkbox to use the same instance pool for worker and driver nodes. NOTE: Kyvos does not perform any heavy operation on the driver node; it is recommended to use a pool of cheaper nodes for the Spark driver, preferably Standard_DS3_v2. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.
Instance Pool Id	Instance Pool Id to be used for worker nodes. If the Use same instance pool for worker and driver option is selected, this pool will also be used for driver nodes. NOTE: Runtime version of the instance pool must be the same as that of the Databricks Cluster ID. You can provide the ID of an existing Instance Pool. See Databricks documentation for details. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.
Driver Instance Pool Id	Instance Pool Id to be used for driver nodes. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.
Spark config	Enter your Spark configuration to fine-tune the Spark job performance. Provide space-separated key-value pair for a property. Multiple properties must also be separated by space. Learn more. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.
Tags (JSON string)	You can add additional tags for the cluster by providing the tags in JSON format. Both the cluster-level tags and those inherited from pools are applied. You cannot add a cluster-specific tag with the same key name as a custom tag inherited from a pool (that is, you cannot override a custom tag that is inherited from the pool). Example: {"key1": "val1","key2": "val2"} Learn more. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.
Cluster log path (DBFS)	You can configure the DBFS location where the system should persist the Spark job logs. If you leave it blank, the system will persist the logs at the dbfs:/cluster-logs location. NOTE: This option is displayed only when you select the Configure Job Cluster checkbox.

Azure JTdCJTIydGl0bGUlMjIlM0ElMjJBenVyZSUyMCUyMiU3RA==

Creating a Process and read connection

From the Toolbox, click Setup, then choose Connections.
From the Actions menu ( ⋮ ) click Add Connection.
Enter a Name for the connection.
From the Category drop-down list, select the Process option.
From the Provider list, select the Databricks option, and provide the following:
1. Databricks Cluster Id: Enter the ID of your Databricks cluster.
  To obtain this ID, click the Cluster Name on the Clusters page in Databricks. The page URL shows <https://<databricks-instance>/#/settings/clusters/<cluster-id>
2. Databricks Service Address: Enter the URL of your Databricks workspace.
3. Databricks Personal Access Token: Enter the personal access token to access and connect to your Databricks workspace. Refer to the Databricks documentation to get your token
To use this connection as a read connection also, select the Is Read checkbox.
Provide the Hive Server JDBC URL. You can find this URL from the Kyvos Manager. Navigate to Hadoop Ecosystem Configuration page > Hadoop Parameters. Copy the value of kyvos.hiveserver2.jdbc.url parameter provided in the format:
jdbc:spark://adb-<Databricks ID>.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/<organization ID>/<Databricks cluster ID>;AuthMech=3;
Ensure that you update the Databricks ID, Organization ID, and Databricks Cluster ID according to the cluster that you are working on.
To use this connection as the default SQL engine, select the Is Default SQL Engine checkbox, and select the SQL engine from Spark, Hive, or Database SQL Warehouse .
See the Provider parameters table for details.
To use the Databricks SQL engine, provide the Server URL and Alternate Server URL.

GCPJTdCJTIydGl0bGUlMjIlM0ElMjJHQ1AlMjIlN0Q=

Creating a Process and read connection

From the Toolbox, click Setup, then choose Connections.
From the Actions menu ( ⋮ ) click Add Connection.
Enter a Name for the connection.
From the Category drop-down list, select the Process option.
From the Provider list, select the Dataproc option.
Provide the Dataproc Cluster Name.
Select the required Hive Authentication from the list. By default, the Is Data Process checkbox is selected.
To use this connection as a read connection also, select the Use as Source checkbox.
Click the Save button.
The system auto-populates the History server Url in the http://[MASTER_NODE_IP]:18080 format according to the provided Dataproc cluster name.
The system auto-populates the Livy server Url in the http://[MASTER_NODE_IP]:8998 format according to the provided Dataproc cluster name.
To use this connection as the default SQL engine, select the Is Default SQL Engine checkbox, and select the SQL engine from Spark or Hive.
See the Provider parameters table for details.
To use the Spark engine, provide the Server URL and Alternate Server URL.
Click the Save button.