Prerequisites for Automated Deployment on Azure with No-Spark

Before you start the automated installation for Kyvos on Azure, ensure you have the following information.

1. Supported only with premium workspace.

2. Supported only with Personal Access Token authentication.

3. On storage, Storage Blob Data Contributor rights are required for the logged-in user.

4. You must have permission to create (and map) Storage credentials and External Locations for the Unity Catalog.

5. Resource Group for all Kyvos resources. We recommend you keep an empty resource group that will only be used for deploying Kyvos resources. The deployment user must have Owner rights on this resource group.

6. If your network resources (for deploying Kyvos) are available in a separate Resource Group (other than the one mentioned in Point 1), create a Custom role for the user deploying the cluster with the following permissions. Refer to the Configuring Roles for Deployment User section for details on creating and assigning roles.
   NOTE: This is not required if you are creating network resources using the Kyvos provided template.

   1. Microsoft.Network/virtualNetworks/subnets/read

   2. Microsoft.Network/virtualNetworks/read

   3. Microsoft.Network/networkSecurityGroups/read

   4. Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action

   5. Microsoft.Network/virtualNetworks/subnets/write

   6. Microsoft.Network/virtualNetworks/subnets/join/action

   7. Microsoft.Network/networkSecurityGroups/join/action

7. Managed Identity for Kyvos resources with the following information:
   NOTE:  As mentioned in the attached Prerequisites sheet, this is optional. It will be created if the value for Enable Managed Identity Creation is set as True in the ARM. 

   1. If you want to use your existing Managed Identity, you will need these details: 

      1. Managed Identity Name

      2. Managed Identity Resource Group Name

   NOTE: If using an existing Managed Identity, ensure that NO permissions are assigned to it.

8. Valid License file for Kyvos.

9. Secret Key to access the Kyvos bundle.

10. Service Endpoints required on Subnet :

    1. Azure Storage (Microsoft.Storage): This model enables you to secure and control the level of access to your storage accounts so that only applications requesting data over the specified set of networks or through the specified set of Azure resources can access a storage account.

    2. Azure Key Vault (Microsoft.KeyVault): The virtual network service endpoints for Azure Key Vault allow you to restrict access to a specified virtual network. The endpoints also allow you to restrict access to a list of IPv4 (internet protocol version 4) address ranges.

    3. Azure App Service (Microsoft.Web): By setting up access restrictions, you can define a priority-ordered allow/deny list that controls network access to your app.

    4. Azure SQL Database (Microsoft.Sql): Security feature that controls whether the server for your databases and elastic pools in Azure SQL Database or for your dedicated SQL pool (formerly SQL DW) databases in Azure Synapse Analytics accepts communications that are sent from particular subnets in virtual networks.

11. Key Vault URL
    If this is not provided, Kyvos will automatically create a new Key Vault for Azure passwords.
    NOTE: You can create your own Key Vault for use with Kyvos. 
    If using an existing Key vault, ensure that the Soft Delete property is enabled, or you can enable it later.

    1. Permissions needed on Key Vault: Assigned managed identity should have permission Secret Permissions (GET, LIST, and SET)

12. In case of Automated deployment, Wizard-based deployment, and/or if using an existing Azure Database for Postgres Flexible Server, ensure that a separate subnet attached to it with delegation (Microsoft.DBforPostgreSWL/flexibleServers and service endpoints Storage, KeyVault, SQL and Web).

13. To use externally created Flexible Server in deployments, use ARM template (FlexibleServerKyvosManagerRepository and FlexibleServerKyvosRepository available in Azure Installation files folder) to create Flexible Server that can be used in the deployments directly. OR, if you create Flexible Server through Microsoft, then you need to complete the following steps. For more information about how to create Flexible Server, refer to Microsoft documentation.

    1. For Kyvos repository

       1. Database name must be delverepo.

       2. Username must be Postgres

       3. Following tags are expected on the external repository:
          UsedBy - Kyvos
          ROLE - DATABASE
          LAYER - Metadata_Storage

    2. For Kyvos Manager repository

       1. Database name must be kmrepo.

       2. Username must be kmdbuser

       3. Following tags are expected on the external repository:
          UsedBy - Kyvos
          ROLE - DATABASE_KM
          LAYER - Metadata_Storage 

14. The Azure logged-in user should have the following rights to create Kyvos resources using ARM templates.

    1. Owner Access on Resource group being used for deployment of Kyvos resources.

    2. Key and Secret Management rights on the Key vault if using an existing Key vault.

15. Networking: Kyvos ARM template will need information about Vnet, Subnet, Network Interface/Security Group that will be used by Kyvos Machines.

    1. Create a Network Interface/Security Group with the following ports opened in Inbound rules. 
       6602, 6903, 6703, 45450, 45460, 6603, 6803, 45440, 6605, 8081, 8080, 45421, 45564, 4000, 7009, 22, 8443, 8444. 9443, 9444.
       To enable Web Portal High Availability,

       1. If using Session Management, you will need 45564 and 4000 ports opened in Inbound rules

       2. If using Azure Load Balancer, you will need port 80. 
          See Ports required for Kyvos for details.

16. SSH Key pair consisting of a private key and a public key.

17. Storage account permission and recommendations:

    1. Managed identity attached to the storage/container should have storage blob data contributor permission.

    2. If the storage account is in a separate Resource Group (different from the one in which the Managed Identity exists), then the Managed Identity should have a Reader role assigned to it at the Storage Account level. This permission is needed by the Kyvos Manager validation framework to check if the Storage Account is accessible or not.

    3. Service principle attached to the Databricks cluster should have storage blob data contributor permission on the storage/container.

    4. Soft deletion property must be disabled.

    5. Storage account must be of type ADLS GEN 2.

18. To access the Usage Dashboard, you need to provide permissions after completing the deployment.

19. For Automated Azure deployment,

    1. Newly created Flexible Server: User provided password will be used for repository. No password change is required.

    2. Existing Flexible Server: Password of the existing repository needs to be provided. No password change is required.

20. If you use an existing Virtual Network, a subnet with at least a /23 mask is required.
    IP Address requirements

26. Kubernetes node pool requirements: Kyvos requires the following two node pools in the AKS (Azure Kubernetes Service) Cluster: 

    1. agentpool 

       1. The mode for this pool will be System. 

       2. This agent pool will be used to run the kyvos-monitoring-server 

       3. The Managed identity attached to this node pool must have permissions to access the Kyvos Storage account. 

       4. This node pool must have taint defined as:   

          • key: "CriticalAddonsOnly” 

          • operator: "Equal" 

          • value: "true" 

          • effect: "NoSchedule" 

    2. userpool 

       1. The mode for this pool will be User. 

       2. This agent pool will be used to run the kyvos-compute-server 

       3. The managed identity attached to this node pool must have permissions to access on the Kyvos Storage account. 

       4. This node must have taint defined as:  

          • key: " ComputeWorkerOnly” 

          • operator: "Equal" 

          • value: "true" 

          • effect: "NoSchedule" 

27. Kubernetes network policy requirements: Kyvos requires two namespaces: kyvos-monitoring and kyvos-compute.  The following ports are required for interactions with important services in these namespaces. 

28. kyvos-monitoring  

    1. Within this namespace, only monitoring services will be operational on the associated pods. The required labels for this service have been applied, specifically kyvos-monitoring-server.  

    2. The network policy for pods with the label kyvos-monitoring-server includes: 

       1. **Ingress:** Ports 80, 443 

       2. **Egress: ** All ports 

29. kyvos-compute 

    1. Within this namespace, compute workers will be operational on the associated pods. The labels required for this service have been applied, specifically kyvos-compute-worker. 

    2. The network policy for pods with the label kyvos-compute-worker includes: 

       1. **Ingress: ** Ports 80, 443, 6903 

       2. **Egress: ** All ports 

30. VPC Requirements: The Kyvos and Kubernetes clusters must be in the same VPC.  

31. CIDR Range for Kubernetes: A subnet with a minimum of a /23 mask is required, and it should have approximately.  

    IP Address Planning

For more details about IP address sizing, see Azure documentation.

26. Kubernetes Subnet Service Endpoints: Following Service Points are required on the subnet:  

    • Microsoft.Storage

    • Microsoft.ContainerRegistry

    • Microsoft.Sql

    • Microsoft.KeyVault 

27. Kyvos APIs and required Kubernetes permissions: Kyvos’ Managed Identity must have a Contributor role on the Kubernetes cluster.  

    1. Monitoring pod API 

    2. Analytical Server  

    3. Kube-Config Credentials (From Analytical server): 

28. Monitoring Pod API: The following Monitoring Pod REST APIs are required:  

    • Azure API for retrieving Kubernetes master details on Azure Kubernetes Service (AKS), including Cluster endpoint information.  

    • REST API for requesting necessary job pods.
      For example, https://34.47.137.13/apis/batch/v1/namespaces/kyvos-compute/jobs  

29. Analytical Server: The following Analytical Server REST APIs are required:  

    • REST API to check the status of the AKS agent pool.  

    • REST API to modify AKS agent pool scaleset.  

    • REST API to check the status of the AKS agent pool scaleset. 

    • REST API to start the AKS cluster. 

    • REST API to get the status of the AKS cluster. 

    • REST API to get instance view of AKS agent pool scaleset. 

    • REST API to get AKS agent pool scaleset information. 

    • REST API to get a list of scaleset from the resource group. 

30. Kube-Config Credentials (From Analytical server): Here is a list of required Kube-Config Credentials (From the Analytical Server) APIs:  

    • Kubernetes Java API for creating deployments and services. 

    • Kubernetes API obtains details on pods, deployments, and services. 

Optional Information

In addition to the prerequisites, you can also keep the following information handy according to your business requirements.

1. Boot Diagnostics Storage Account Uri: This is needed to store the Bootup logs of the VM. In case the value is empty, the boot diagnostics feature will be disabled for all Kyvos VMs.

2. Log Analytics Workspace Name and  Resource Group Name: This is needed for enabling Log Analytics virtual machine extension for VMs used for monitoring Azure VM.

3. Shared Image Gallery Information: If you want to use your hardened images as a base OS for  Kyvos  VMs, then you will need the information for the following parameters:

   1. Gallery Resource Group Name: Name of the Resource Group in which the Gallery resides.

   2. Gallery Subscription ID: Subscription ID in which Gallery resides.

   3. Gallery Name: Name of the Shared Image Gallery. An Azure image gallery is a repository for managing and sharing custom images. An image source can be an existing Azure VM.

   4. Gallery Image Definition Name:  Name of the Image Definition. Image definitions are created within a gallery and carry information about the image and requirements for using it internally. This includes whether the image is Windows or Linux, release notes, and minimum and maximum memory requirements. It is a definition of a type of image.

   5. Gallery Image Version Name:  Name of the Image Version in the <MajorVersion>.<MinorVersion>.<Patch> format.

4. Azure Function Name: If you want to use pre-created functions, then you can either provide the name of the function at the time of creating Kyvos resources through Kyvos Manager (Wizard-based deployment), or you can create the function externally (from the Azure portal), using the azure_functions.json  template file or azure_functions_secure.json file (for enhanced security) available in the Azure Installation Files folder. If upgrading from an older version (prior to 2021.3), you can use the steps mentioned in the section Post-upgrade Steps.