Azure Key Vault Configuration for Kyvos

Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Azure Key Vault is a cloud service that provides a secure store for secrets. You can securely store keys, passwords, certificates, and other secrets in the vault.

You can create and manage the key vaults through the Azure portal.

Creating a vault

Create a vault as explained at Quickstart - Create an Azure Key Vault with the Azure portal | Microsoft Learn

Note the Vault Name and Vault URL for use later.

Access configurations for Key Vault

Once you have created the Key Vault, navigate to the Access Configuration page, and perform the following steps:

From the Permission Model, select the option.
From the Resource Access, choose the following permissions (as shown in the figure below).
1. Azure Virtual Machines for deployment
2. Azure Resource Manager for template deployment
3. Azure Disk Encryption for volume encryption

Adding secret to Key Vault

To add a secret to the vault, you need to perform a few additional steps. In this case, we add a password that could be used by an application. The password is called ExamplePassword, and we store the value of hVFkk965BuUv in it.

On the Key Vault properties page, select Secrets.
Click on Generate/Import.
On the Create a secret screen, select the following values:
1. Upload options: Manual
2. Name: ExamplePassword
3. Value: hVFkk965BuUv
Leave the other values to their defaults. Click Create.
By clicking the Show Secret Value button in the right pane, you can see the hidden value.

Note

Please note down the Name of Secret given in the <secret-name> as you will need it later.

Create an Azure Key Vault-backed secret scope

Verify that you have Contributor permission on the Azure Key Vault instance that you want to use to back the secret scope.
Go to https://<your_azure_databricks_url>#secrets/createScope (for example, https://westus.azuredatabricks.net#secrets/createScope ). This URL is case sensitive; the scope in createScope must be uppercase.
Enter the name of the secret scope. Secret scope names are case insensitive.
Use the Manage Principal drop-down to specify whether All Users have to MANAGE permission for this secret scope or only the Creator of the secret scope (that is to say, you).
1. MANAGE permission allows users to read and write to this secret scope, and, in the case of accounts on the Azure Databricks Premium Plan, to change permissions for the scope.
2. Your account must have the Azure Databricks Premium Plan for you to be able to select Creator . This is the recommended approach: grant MANAGE permission to the Creator When you create the secret scope, and then assign more granular access permissions after you have tested the scope. For example workflow, see Secret workflow example.
3. If your account has the Standard Plan, you must set the MANAGE permission to the “All Users” group. If you select Creator here, you will see an error message when you try to save the scope.
4. For more information about the MANAGE permission, see Secret access control.
Enter the DNS Name (for example, https://databrickskv.vault.azure.net/ ) and Resource ID, for example: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/databricks-rg/providers/Microsoft.KeyVault/vaults/databricksKV.
These properties are available from the Properties tab of an Azure Key Vault in your Azure portal.
1. Click the Create button.
2. Use the Databricks CLI Databricks secrets list-scopes command to verify that the scope was created successfully.

Note

Please note down the Name of the Secret given in the <secret-name> as you will need it later.

Using Key Vault Secret in Azure Databricks

The Key Vault secret value is referred to in Databricks using the below syntax.
{{secrets/<scope-name>/<secret-name>}} .

The value must start with {{secrets/ and end with }} . The variable portions of the property or environment variable are:

<scope-name> : The name of the scope in which the secret is associated.
<secret-name> : The unique name of the secret in the scope.

Use the secret value with Spark property as: spark.hadoop.fs.azure.account.oauth2.client.secret {{secrets /<scope-name>/<secret-name>}}