Azure Key Vault Configuration for Kyvos
Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace
Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)
Azure Key Vault is a cloud service that provides a secure store for secrets. You can securely store keys, passwords, certificates, and other secrets in the vault.
You can create and manage the key vaults through the Azure portal.
Creating a vault
Create a vault as explained at Quickstart - Create an Azure Key Vault with the Azure portal | Microsoft Learn
Note the Vault Name and Vault URL for use later.
Access configurations for Key Vault
Once you have created the Key Vault, navigate to the Access Configuration page, and perform the following steps:
From the Permission Model, select the option.
From the Resource Access, choose the following permissions (as shown in the figure below).
Azure Virtual Machines for deployment
Azure Resource Manager for template deployment
Azure Disk Encryption for volume encryption
Adding secret to Key Vault
To add a secret to the vault, you need to perform a few additional steps. In this case, we add a password that could be used by an application. The password is called ExamplePassword, and we store the value of hVFkk965BuUv in it.
On the Key Vault properties page, select Secrets.
Click on Generate/Import.
On the Create a secret screen, select the following values:
Upload options: Manual
Name: ExamplePassword
Value: hVFkk965BuUv
Leave the other values to their defaults. Click Create.
By clicking the Show Secret Value button in the right pane, you can see the hidden value.
Note
Please note down the Name of Secret given in the <secret-name> as you will need it later.
Create an Azure Key Vault-backed secret scope
Verify that you have Contributor permission on the Azure Key Vault instance that you want to use to back the secret scope.
Go to https://<your_azure_databricks_url>#secrets/createScope (for example, https://westus.azuredatabricks.net#secrets/createScope ). This URL is case sensitive; the scope in createScope must be uppercase.
Enter the name of the secret scope. Secret scope names are case insensitive.
Use the Manage Principal drop-down to specify whether All Users have to MANAGE permission for this secret scope or only the Creator of the secret scope (that is to say, you).
MANAGE permission allows users to read and write to this secret scope, and, in the case of accounts on the Azure Databricks Premium Plan , to change permissions for the scope.
Your account must have the Azure Databricks Premium Plan for you to be able to select Creator . This is the recommended approach: grant MANAGE permission to the Creator When you create the secret scope, and then assign more granular access permissions after you have tested the scope. For example workflow, see Secret workflow example .
If your account has the Standard Plan, you must set the MANAGE permission to the “All Users” group. If you select Creator here, you will see an error message when you try to save the scope.
For more information about the MANAGE permission, see Secret access control .
Enter the DNS Name (for example, https://databrickskv.vault.azure.net/ ) and Resource ID, for example: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/databricks-rg/providers/Microsoft.KeyVault/vaults/databricksKV.
These properties are available from the Properties tab of an Azure Key Vault in your Azure portal.
Click the Create button.
Use the Databricks CLI Databricks secrets list-scopes command to verify that the scope was created successfully.
Note
Please note down the Name of the Secret given in the <secret-name> as you will need it later.
Using Key Vault Secret in Azure Databricks
The Key Vault secret value is referred to in Databricks using the below syntax.
{{secrets/<scope-name>/<secret-name>}} .
The value must start with {{secrets/ and end with }} . The variable portions of the property or environment variable are:
<scope-name> : The name of the scope in which the secret is associated.
<secret-name> : The unique name of the secret in the scope.
Use the secret value with Spark property as: spark.hadoop.fs.azure.account.oauth2.client.secret {{secrets /<scope-name>/<secret-name>}}
Copyright Kyvos, Inc. All rights reserved.