Versions Compared

Old Version 2

changes.mady.by.user Sarabjeet Kaur

Saved on

compared with

New Version Current

changes.mady.by.user Sarabjeet Kaur

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Applies to: (tick) Kyvos Enterprise  (error) Kyvos Cloud (SaaS on AWS) (error) Kyvos AWS Marketplace

(error) Kyvos Azure Marketplace   (error) Kyvos GCP Marketplace (error) Kyvos Single Node Installation (Kyvos SNI)

...

Before you begin

In addition to the prerequisites, please ensure the following settings are enabled on your GCP project.

  1. Project Billing: For this, search Billing on your Google Cloud project.

    1. Click Link a Billing Account, and configure the billing information.
      Once your billing is enabled, you will see an estimate as shown in the following figure.

  2. Cloud Deployment Manager V2 API: Search Cloud Deployment Manager V2 API on your project and click the Enable button.
    Once the API is enabled, the corresponding API Enabled status is displayed, as shown in the following figure.

  3. Compute Engine APIs: Search for Compute Engine APIs on your project, and click the Enable button.
    Once the API is enabled, the corresponding status is displayed, as shown in the following figure.

  4. Cloud Resource Manager API: Search for Cloud Resource Manager API on your project, and click the Enable button.
    Once the API is enabled, the corresponding API Enabled status is displayed, as shown in the following figure.

  5. Enable the following APIs on your project. Refer to the GCP documentation for details.

    1. Cloud Functions

    2. Cloud Build

    3. Cloud Scheduler

  6. Create an App Engine project, and select the region where you want to deploy your resources.

  7. To the default Google APIs Service Agent service account, add the storage.buckets.get role. This is required to delete deployment through the Deployment Manager.

  8. Kubernetes Engine API: Search for Kubernetes Engine API on your project and click the Enable button. Once the API is enabled, the API Enabled status is displayed.

    image-20240613-124249.pngImage Added

...

Creating resources using script

...

paramsJTdCJTIyZ2VuZXJhbCUyMiUzQSU3QiUyMnRhYlNwYWNpbmclMjIlM0EwJTJDJTIydGFiV2lkdGglMjIlM0ExMDAlMkMlMjJ0YWJIZWlnaHQlMjIlM0E1MCUyQyUyMmRpcmVjdGlvbiUyMiUzQSUyMmhvcml6b250YWwlMjIlN0QlMkMlMjJjb250ZW50JTIyJTNBJTdCJTIyYmFja2dyb3VuZENvbG9yJTIyJTNBJTdCJTIyY29sb3IlMjIlM0ElMjIlMjNmZmYlMjIlN0QlMkMlMjJib3JkZXIlMjIlM0ElN0IlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyd2lkdGglMjIlM0ExJTJDJTIydG9wJTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmNvbG9yJTIyJTNBJTdCJTIybGlnaHQlMjIlM0ElMjIlMjNjY2NlY2YlMjIlN0QlN0QlMkMlMjJwYWRkaW5nJTIyJTNBJTdCJTIydG9wJTIyJTNBMTAlMkMlMjJyaWdodCUyMiUzQTEwJTJDJTIyYm90dG9tJTIyJTNBMTAlMkMlMjJsZWZ0JTIyJTNBMTAlN0QlN0QlMkMlMjJhY3RpdmUlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzZjU4MjI3JTIyJTdEJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzMDAwMDAwJTIyJTdEJTJDJTIydGV4dEFsaWduJTIyJTNBJTIybGVmdCUyMiUyQyUyMmZvbnRXZWlnaHQlMjIlM0ElMjJib2xkJTIyJTdEJTdEJTJDJTIyaG92ZXIlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSUyMiUyM2RmZTFlNiUyMiU3RCUyQyUyMnRleHQlMjIlM0ElN0IlMjJmb250U2l6ZSUyMiUzQTE4JTJDJTIyY29sb3IlMjIlM0ElMjIlMjM1ZTZjODQlMjIlMkMlMjJ0ZXh0QWxpZ24lMjIlM0ElMjJsZWZ0JTIyJTJDJTIyZm9udFdlaWdodCUyMiUzQSUyMmxpZ2h0ZXIlMjIlN0QlN0QlMkMlMjJpbmFjdGl2ZSUyMiUzQSU3QiUyMmJhY2tncm91bmRDb2xvciUyMiUzQSU3QiUyMmNvbG9yJTIyJTNBJTIyJTIzZjRmNWY3JTIyJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSUyMiUyMzVlNmM4NCUyMiUyQyUyMnRleHRBbGlnbiUyMiUzQSUyMmxlZnQlMjIlMkMlMjJmb250V2VpZ2h0JTIyJTNBJTIybGlnaHRlciUyMiU3RCUyQyUyMmJvcmRlciUyMiUzQSU3QiUyMnRvcCUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJ3aWR0aCUyMiUzQTElMkMlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyY29sb3IlMjIlM0ElN0IlMjJsaWdodCUyMiUzQSUyMiUyM2NjY2VjZiUyMiU3RCU3RCU3RCU3RA==

...

Anchor
creatingscript
creatingscript

  1. Download the gcp.tar file from the GCP Installation Files folder on your workstation.

  2. On your workstation, install the gcloud command-line tool.

  3. Configure the gcloud command-line tool to use your project using the following command.
    gcloud config set project [MY_PROJECT]
    Here, replace [MY_PROJECT] with your project ID.

  4. Copy the script tar file and untar it.

  5. Update the following parameters in the kyvos-template.yaml (provided in the gcp.tar) as per your business requirement.

  6. Enter details as:

To create firewall rules, set the value of this parameter to true
Aura tab collection
paramsJTdCJTIyZ2VuZXJhbCUyMiUzQSU3QiUyMnRhYlNwYWNpbmclMjIlM0EwJTJDJTIydGFiV2lkdGglMjIlM0ExMDAlMkMlMjJ0YWJIZWlnaHQlMjIlM0E1MCUyQyUyMmRpcmVjdGlvbiUyMiUzQSUyMmhvcml6b250YWwlMjIlN0QlMkMlMjJjb250ZW50JTIyJTNBJTdCJTIyYmFja2dyb3VuZENvbG9yJTIyJTNBJTdCJTIyY29sb3IlMjIlM0ElMjIlMjNmZmYlMjIlN0QlMkMlMjJib3JkZXIlMjIlM0ElN0IlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyd2lkdGglMjIlM0ExJTJDJTIydG9wJTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmNvbG9yJTIyJTNBJTdCJTIybGlnaHQlMjIlM0ElMjIlMjNjY2NlY2YlMjIlN0QlN0QlMkMlMjJwYWRkaW5nJTIyJTNBJTdCJTIydG9wJTIyJTNBMTAlMkMlMjJyaWdodCUyMiUzQTEwJTJDJTIyYm90dG9tJTIyJTNBMTAlMkMlMjJsZWZ0JTIyJTNBMTAlN0QlN0QlMkMlMjJhY3RpdmUlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzZjU4MjI3JTIyJTdEJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzMDAwMDAwJTIyJTdEJTJDJTIydGV4dEFsaWduJTIyJTNBJTIybGVmdCUyMiUyQyUyMmZvbnRXZWlnaHQlMjIlM0ElMjJib2xkJTIyJTdEJTdEJTJDJTIyaG92ZXIlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSUyMiUyM2RmZTFlNiUyMiU3RCUyQyUyMnRleHQlMjIlM0ElN0IlMjJmb250U2l6ZSUyMiUzQTE4JTJDJTIyY29sb3IlMjIlM0ElMjIlMjM1ZTZjODQlMjIlMkMlMjJ0ZXh0QWxpZ24lMjIlM0ElMjJsZWZ0JTIyJTJDJTIyZm9udFdlaWdodCUyMiUzQSUyMmxpZ2h0ZXIlMjIlN0QlN0QlMkMlMjJpbmFjdGl2ZSUyMiUzQSU3QiUyMmJhY2tncm91bmRDb2xvciUyMiUzQSU3QiUyMmNvbG9yJTIyJTNBJTIyJTIzZjRmNWY3JTIyJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSUyMiUyMzVlNmM4NCUyMiUyQyUyMnRleHRBbGlnbiUyMiUzQSUyMmxlZnQlMjIlMkMlMjJmb250V2VpZ2h0JTIyJTNBJTIybGlnaHRlciUyMiU3RCUyQyUyMmJvcmRlciUyMiUzQSU3QiUyMnRvcCUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJ3aWR0aCUyMiUzQTElMkMlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyY29sb3IlMjIlM0ElN0IlMjJsaWdodCUyMiUzQSUyMiUyM2NjY2VjZiUyMiU3RCU3RCU3RCU3RA==
Aura tab
summaryKyvos resources and Dataproc for Multiple build connections
paramsJTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMGFuZCUyMERhdGFwcm9jJTIwZm9yJTIwTXVsdGlwbGUlMjBidWlsZCUyMGNvbm5lY3Rpb25zJTIyJTdE

Parameter

Description

createVPC

Set the value as true to create a new VPC.

customPrefixNameVPC

A custom prefix name for a VPC. 

customPrefixNameSubNetwork

A custom prefix name for the subnetwork. 

ipCidrRange

Enter the VPC CIDR range in which VPC will be created. 

vpc

Enter the VPC within which you want to deploy the resources.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

vpcProjectId

Enter the Project ID of your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

createNetworkFirewall

subnetwork

Enter the subnetwork within your VPC.

NOTE:

If the value of createVPC is set to true, firewall rules will be created unconditionally. 

subnetwork

Enter the subnetwork within your VPC.

NOTE:

You must leave this field blank if the parameter createVPC is set to true. 

enableVpcConnector

Keep

createNetworkFirewall

To create firewall rules, set the value of this parameter

as false if

to false.

NOTE: If the value of

the parameter (

createVPC

)

is set to true, firewall rules will be created unconditionally. 

vpcConnectorName

gkeSubnetName

Enter the name of

the VPC Connector to be used with GCP functions. 

kmCount

The number of Kyvos Manager instances to be launched.

kmInstanceType

Instance type of Kyvos Manager (n2-standard-4). 

kmVolumeSizeGB

Size of the disk to be attached to the Kyvos Manager.

kmBootScript startup-script

startup-script-km.sh

kmSourceImage

Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07

RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

webServerPublicIP

Change the value to true if public IP for the Web Server is needed.
NOTE: Change the value to true if the value of createVPC is true

kmVolumeType

Type of the disk for KM (pd-ssd).

hostNameBasedDeployment

Change the value to true to use the hostname for the cluster deployment.

qeCount

The number of instances to be used as query engines.

qeInstanceType

Instance type of query engine (n2-highmem-4).

qeDataVolumeSizeGB

an existing Subnet in which you want to deploy GKE Cluster. If left blank subnetwork name will be used.

secondaryRangeName1

Enter the Secondary IPv4 ranges name for GKE Cluster creation.

NOTE: This must be preconfigured if using an existing VPC.
The range should have a minimum masking of /22.

For more information, see Google documentation.

secondaryRangeName2

Enter the Secondary IPv4 ranges name for GKE Cluster creation.

NOTE: This must be preconfigured if using an existing VPC.
The range should have a minimum masking of /22.

For more information, see Google documentation.

enableVpcConnector

Keep the value of this parameter as false if the value of the parameter (createVPC) is true. 

vpcConnectorName

Enter the name of the VPC Connector to be used with GCP functions. 

kmCount

The number of Kyvos Manager instances to be launched.

kmInstanceType

Instance type of Kyvos Manager (n2-standard-4). 

kmVolumeSizeGB

Size of the disk to be attached

with query engines.

qeCacheVolumeSizeGB

Size of the disk

to

be attached for

the

cache.

qeCacheVolumeCount

The number of disks to be attached to the cache.

qeCacheVolumeType

Type of the disk for cache storage (pd-ssd).

biCount

Enter the number of instances to be used as the BI server.

biInstanceType

Instance type of BI Server (n2-standard-8).

biBootScript

Kyvos Manager.

kmBootScript startup-script

startup-script-

bi

km.sh

sourceImage

kmSourceImage

Ubuntu

:

- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-

image-2023-11-07

2024-2

RHEL

:

- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

biVolumeCount

The number of disks to be attached to the BI Server.

biVolumeSizeGB

Size of the disk to be attached to the BI Server.

biVolumeType

webServerPublicIP

Change the value to true if public IP for the Web Server is needed.
NOTE: Change the value to true if the value of createVPC is true

kmVolumeType

Type of the disk for

BI server

KM (pd-ssd).

createGcpFunctions

hostNameBasedDeployment

Set

Change the value

as

to true to

configure GCP Functions in Kyvos

use the hostname for the cluster deployment.

createDataProc

Enter true to create Dataproc. 

enableComponentGateway

Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.

sharedDataprocCluster

Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.  

dataProcNetworkTags

Provide a list of comma-separated network tags to be added to the Dataproc cluster.

Example: dataProcNetworkTags : ["abc","xyz"]

enableSshFlag

Set the value to true to enable SSH to the Dataproc cluster.

enableLivy

Set the value of Livy to True if using Dataproc version 2.1.11-debian11.

masterInstanceCount

The number of master nodes. For example, 1 

masterInstanceType

Instance type of master node (n2-highmem-4)

masterInstanceVolumeType

Type of the disk for master node (pd-ssd)

workerInstanceCount

The number of worker instances.

workerInstanceType

Instance type of worker node (n2-highmem-8)

workerInstanceVolumeType

Type of the disk for worker node (pd-ssd)

enableDataProcMetastore

Set the value as true to allow external Dataproc metastore.
NOTE: Existing metastore is not supported if the value of the createVPC is true

dataProcMetastoreProjectId

If enable DataProcMetastore is set as true, provide the name of the metastore project ID.

dataProcMetaStoreName

Provide the name of the metastore name.

dataProcVersion

Supported version is 2.1.11-debian11

enableAutoScaling

Set the value as true to enable the autoscaling of cluster nodes.

existingAutoScalingPolicyName

Provide the name of the existing autoscaling property, if any.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMinInstanceCount

Specify the number of minimum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMaxInstanceCount

Specify the number of maximum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

existingDataprocClusterName

Enter the name of the existing Dataproc cluster. 

NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. 

sshPrivateKeyDataproc

The private key of existing Dataproc.

NOTE: The private key should be base64 encoded

dataprocUsername

Name of the user. 

createServiceAccount

 Change the value to false if you want to use the existing Service Account.

serviceAccountName

Enter the service account name to be attached to all Kyvos Virtual Machines. 

secretManagerName

Provide the name of the existing Secret Manager.

secretManagerProjectId

Provide the name of the Project ID in which the existing secret manager is created.

customPrefixNameDataproc

A prefix is to be added before the name of Dataproc.

customPrefixNameBI

A prefix is to be added before the name of BI virtual machines.

customPrefixNameBIDisk

A prefix is to be added before the name of BI Disks.

customPrefixNameQE

A prefix is to be added before the name of Query Engine virtual machines.

customPrefixNameQEDisk

A prefix is to be added before the name of Query Engine disks.

customPrefixNameKM

A prefix is to be added before the name of the Kyvos Manager virtual machine. 

customPrefixNameKMDataDisk

A prefix is to be added before the name of the Kyvos Manager disk.

zone

The zone in which you want to deploy the resources.

bucketName

Enter the bucket name (if the bucket is not present, a new bucket would be created).

kyvosWorkDir

Provide the path of the Kyvos work directory.

kyvosClusterName

Name of the Kyvos cluster to be deployed.

bundleAccessKey

Key to access Kyvos bundle.

bundleSecretKey

The secret key for Kyvos bundle.

sshPublicKey

Text of the SSH public key for authentication.

sshPrivateKey

Enter the SSH private Key text of the pem file.

NOTE: The text must be in base64 encoded.

kyvosLicenseFileValue

Enter a valid Kyvos license key.

NOTE: It should be base64 encoded

additionalLabels

Labels to be added to the resources. 

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

Possible values in volume type fields:

For SSD type disk: pd-ssd

For standard disk: pd-standard

Aura tab
summaryKyvos resources and Dataproc for Multiple build connections
paramsJTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMGFuZCUyMERhdGFwcm9jJTIwZm9yJTIwTXVsdGlwbGUlMjBidWlsZCUyMGNvbm5lY3Rpb25zJTIyJTdE

  1. Download the gcp.tar file from the GCP Installation Files folder on your workstation.

  2. On your workstation, install the gcloud command-line tool.

  3. Configure the gcloud command-line tool to use your project using the following command.

gcloud config set project [MY_PROJECT]
Here, replace [MY_PROJECT] with your project ID.

  1. Copy the script tar file and untar it.

  2. Update the following parameters in the kyvos-template.yaml (provided in the gcp.tar) as per your business requirement.

Parameter

Description

createVPC

Set the value as true to create a new VPC.

customPrefixNameVPC

A custom prefix name for a VPC. 

customPrefixNameSubNetwork

A custom prefix name for the subnetwork. 

ipCidrRange

Enter the VPC CIDR range in which VPC will be created. 

vpc

Enter the VPC within which you want to deploy the resources.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

vpcProjectId

Enter the Project ID of your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

createNetworkFirewall

To create firewall rules, set the value of this parameter to true.

NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally. 

subnetwork

Enter the subnetwork within your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

enableVpcConnector

Keep the value of this parameter as false if the value of the parameter (createVPC) is true. 

vpcConnectorName

Enter the name of the VPC Connector to be used with GCP functions. 

kmCount

The number of Kyvos Manager instances to be launched.

kmInstanceType

Instance type of Kyvos Manager (n2-standard-4). 

kmVolumeSizeGB

Size of the disk to be attached to the Kyvos Manager.

kmBootScript

startup-script

startup-script-km.sh

kmSourceImage

Enter the following path:
Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07

RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

webServerPublicIP

Change the value to true if public IP for the Web Server is needed.
NOTE: Change the value to true if the value of createVPC is true

kmVolumeType

Type of the disk for KM (pd-ssd).

hostNameBasedDeployment

Change the value to true to use the hostname for the cluster deployment

qeCount

The number of instances to be used as query engines.

qeInstanceType

Instance type of query engine (n2-highmem-4).

qeDataVolumeSizeGB

Size of the disk to be attached with query engines.

qeCacheVolumeSizeGB

Size of the disk to be attached for the cache.

qeCacheVolumeCount

The number of disks to be attached to the cache.

qeCacheVolumeType

Type of the disk for cache storage (pd-ssd).

biCount

Enter the number of instances to be used as the BI server.

biInstanceType

Instance type of BI Server (n2-standard-8).

biBootScript

startup-script-bi.sh

sourceImage

Enter the following path:
Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07

RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

biVolumeCount

The number of disks to be attached to the BI Server.

biVolumeSizeGB

Size of the disk to be attached to the BI Server.

biVolumeType

Type of the disk for BI server (pd-ssd)

createGcpFunctions

Set the value as true to configure GCP Functions in Kyvos.

createDataProc

Enter true or false as per your requirement. 

enableComponentGateway

Set enableComponentGateway to true to get publicly accessible URL for Dataproc.

sharedDataprocCluster

Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.

Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.  

dataProcNetworkTags

Provide a list of comma-separated network tags to be added on Dataproc cluster.
Example: dataProcNetworkTags : ["abc","xyz"]

enableSshFlag

Set the value to true to enable SSH to the Dataproc cluster.

enableLivy

Set the value of Livy to True if using Dataproc version 2.1.11-debian11.

masterInstanceCount

The number of master nodes. For example, 1 

masterInstanceType

Instance type of master node (n2-highmem-4)

masterInstanceVolumeType

Type of the disk for master node (pd-ssd)

workerInstanceCount

The number of worker instances.

workerInstanceType

Instance type of worker node (n2-highmem-8)

workerInstanceVolumeType

Type of the disk for worker node (pd-ssd is recommended)

enableDataProcMetastore

Set the value as true to allow external Dataproc metastore.
NOTE: Existing metastore is not supported if the value of the createVPC is true

dataProcMetastoreProjectId

If enable DataProcMetastore is set as true, provide the name of the metastore project ID.

dataProcMetaStoreName

Provide the name of the metastore name.

dataProcVersion

Supported version:

  •  2.1.11-debian11

enableAutoScaling

Set the value as true to enable the autoscaling of cluster nodes.

existingAutoScalingPolicyName

Provide the name of the existing autoscaling property, if any.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMinInstanceCount

Specify the number of minimum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMaxInstanceCount

Specify the number of maximum worker instances to be kept running while scaling.

NOTE: Use this configuration only if enableAutoScaling is set as true.

existingDataprocClusterName

Enter the name of the existing Dataproc cluster. 

NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. 

sshPrivateKeyDataproc

The private key of existing Dataproc.

NOTE: The private key should be base64 encoded

dataprocUsername

Name of the existing user. 

createServiceAccount

Change the value to false if you want to use the existing Service Account. 

serviceAccountName

Enter the service account name to be attached to all Kyvos Virtual Machines. 

secretManagerName

Provide the name of the existing Secret Manager.

secretManagerProjectId

Provide the name of the Project ID in which the existing secret manager is created.

customPrefixNameDataproc

A prefix is to be added before the name of Dataproc.

customPrefixNameBI

A prefix is to be added before the name of BI virtual machines.

customPrefixNameBIDisk

A prefix is to be added before the name of BI Disks.

customPrefixNameQE

A prefix is to be added before the name of Query Engine virtual machines.

customPrefixNameQEDisk

A prefix is to be added before the name of Query Engine disks.

customPrefixNameKM

A prefix is to be added before the name of the Kyvos Manager virtual machine. 

customPrefixNameKMDataDisk

A prefix is to be added before the name of the Kyvos Manager disk.

zone

The zone in which you want to deploy the resources.

bucketName

Enter the bucket name (if the bucket is not present, a new bucket would be created).

kyvosWorkDir

Provide the path of the Kyvos work directory.

kyvosClusterName

Name of the Kyvos cluster to be deployed.

bundleAccessKey

Key to access Kyvos bundle.

bundleSecretKey

The secret key for Kyvos bundle.

sshPublicKey

Text of the SSH public key for authentication.

sshPrivateKey

Enter the SSH private Key text of the pem file.

NOTE: The text must be in base64 encoded.

kyvosLicenseFileValue

Enter a valid Kyvos license key.

NOTE: It should be base64 encoded

additionalLabels

Labels to be added to the resources. 

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

Possible values in volume type fields:

For SSD type disk: pd-ssd

For standard disk: pd-standard

To create Dataproc for use with multiple build connections, use the multiple-build-cluster.tar file from the GCP Installation Files folder. In this case, use the dataproc-template.yaml file to create Dataproc.

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

All the build clusters must use the same GCP Dataproc metastore.

In the dataproc-template.yaml file, provide information as:

  1. For bucketName parameter, mention the bucket name used in the Kyvos deployment.

  2. For serviceAccountName parameter, mention the service account used in Kyvos resources.

  3. For kyvosWorkDir parameter, provide the working directory of Kyvos.

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

After filling all the required details in the dataproc-template.yaml file, run the following command:

gcloud deployment-manager deployments create <Name of resources> --config dataproc-template.yaml

Aura tab
summaryKyvos resources and Dataproc for Kubernetes
paramsJTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMGFuZCUyMERhdGFwcm9jJTIwZm9yJTIwS3ViZXJuZXRlcyUyMiU3RA==

Parameter

Description

createVPC

Set the value as true to create a new VPC.

customPrefixNameVPC

A custom prefix name for a VPC. 

customPrefixNameSubNetwork

A custom prefix name for the subnetwork. 

ipCidrRange

Enter the VPC CIDR range in which VPC will be created. 

vpc

Enter the VPC within which you want to deploy the resources.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

vpcProjectId

Enter the Project ID of your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

subnetwork

Enter the subnetwork within your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

createNetworkFirewall

To create firewall rules, set the value of this parameter to false.

NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally. 

gkeSubnetName

Enter the name of existing Subnet in which you want to deploy GKE Cluster. If left blank subnetwork name will be used.

secondaryRangeName1

Enter the Secondary IPv4 ranges name for GKE Cluster creation.

secondaryRangeName2

Enter the Secondary IPv4 ranges name for GKE Cluster creation.

NOTE: This must be preconfigured if using an existing VPC.
The range should havea minimum masking of /22.

enableVpcConnector

Keep the value of this parameter as false if the value of the parameter (createVPC) is true. 

vpcConnectorName

Enter the name of the VPC Connector to be used with GCP functions. 

kmCount

The number of Kyvos Manager instances to be launched.

kmInstanceType

Instance type of Kyvos Manager (n2-standard-4). 

kmVolumeSizeGB

Size of the disk to be attached to the Kyvos Manager.

kmBootScript startup-script

startup-script-km.sh

kmSourceImage

Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07

RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

webServerPublicIP

Change the value to true if public IP for the Web Server is needed.
NOTE: Change the value to true if the value of createVPC is true

kmVolumeType

Type of the disk for KM (pd-ssd).

hostNameBasedDeployment

Change the value to true to use the hostname for the cluster deployment.

qeCount

The number of instances to be used as query engines.

qeInstanceType

Instance type of query engine (n2-highmem-4).

qeDataVolumeSizeGB

Size of the disk to be attached with query engines.

qeCacheVolumeSizeGB

Size of the disk to be attached for the cache.

qeCacheVolumeCount

The number of disks to be attached to the cache.

qeCacheVolumeType

Type of the disk for cache storage (pd-ssd).

biCount

Enter the number of instances to be used as the BI server.

biInstanceType

Instance type of BI Server (n2-standard-8).

biBootScript

startup-script-bi.sh

sourceImage

Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07

RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

biVolumeCount

The number of disks to be attached to the BI Server.

biVolumeSizeGB

Size of the disk to be attached to the BI Server.

biVolumeType

Type of the disk for BI server (pd-ssd)

createGcpFunctions

Set the value as true to configure GCP Functions in Kyvos.

createDataProc

Enter true to create Dataproc. 

dataprocMetastoreURI

Enter the Metastore URI if you want to deploy Kyvos with no Spark configuration.

createGKE

Enter the value as True or False.

  • True: To create Kubernetes cluster.

gkeWorkerInitialNodeCount

Enter the initial worker node count for the Kubernetes cluster.

NOTE: The default value is 1.

gkeWorkerInstancetype

Enter the worker node instance type for the Kubernetes cluster.

NOTE: n2-highmem-8 is the minimum configuration. Instance type smaller than this aren't supported.

minWorkerNodeCount

Enter the minimum worker node count.

maxWorkerNodeCount

Enter the maximum worker node count.

createDataProc

Enter the value as True or False.

  • True: If you want to deploy Kyvos with Spark Configuration.

enableComponentGateway

Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.

sharedDataprocCluster

Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.  

dataProcNetworkTags

Provide a list of comma-separated network tags to be added to the Dataproc cluster.

Example: dataProcNetworkTags : ["abc","xyz"]

enableSshFlag

Set the value to true to enable SSH to the Dataproc cluster.

enableLivy

Set the value of Livy to True if using Dataproc version 2.1.11-debian11.

enableTrino

masterInstanceCount

The number of master nodes. For example, 1 

masterInstanceType

Instance type of master node (n2-highmem-4)

masterInstanceVolumeType

Type of the disk for master node (pd-ssd)

workerInstanceCount

The number of worker instances.

workerInstanceType

Instance type of worker node (n2-highmem-8)

workerInstanceVolumeType

Type of the disk for worker node (pd-ssd)

enableDataProcMetastore

Set the value as true to allow external Dataproc metastore.
NOTE: Existing metastore is not supported if the value of the createVPC is true

dataProcMetastoreProjectId

If enable DataProcMetastore is set as true, provide the name of the metastore project ID.

dataProcMetaStoreName

Provide the name of the metastore name.

dataProcVersion

Supported version is 2.1.11-debian11

enableAutoScaling

Set the value as true to enable the autoscaling of cluster nodes.

existingAutoScalingPolicyName

Provide the name of the existing autoscaling property, if any.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMinInstanceCount

Specify the number of minimum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMaxInstanceCount

Specify the number of maximum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

existingDataprocClusterName

Enter the name of the existing Dataproc cluster. 

NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. 

sshPrivateKeyDataproc

The private key of existing Dataproc.

NOTE: The private key should be base64 encoded

dataprocUsername

Name of the user. 

createServiceAccount

 Change the value to false if you want to use the existing Service Account.

serviceAccountName

Enter the service account name to be attached to all Kyvos Virtual Machines. 

secretManagerName

Provide the name of the existing Secret Manager.

secretManagerProjectId

Provide the name of the Project ID in which the existing secret manager is created.

customPrefixNameDataproc

A prefix is to be added before the name of Dataproc.

customPrefixNameBI

A prefix is to be added before the name of BI virtual machines.

customPrefixNameBIDisk

A prefix is to be added before the name of BI Disks.

customPrefixNameQE

A prefix is to be added before the name of Query Engine virtual machines.

customPrefixNameQEDisk

A prefix is to be added before the name of Query Engine disks.

customPrefixNameKM

A prefix is to be added before the name of the Kyvos Manager virtual machine. 

customPrefixNameKMDataDisk

A prefix is to be added before the name of the Kyvos Manager disk.

zone

The zone in which you want to deploy the resources.

bucketName

Enter the bucket name (if the bucket is not present, a new bucket would be created).

kyvosWorkDir

Provide the path of the Kyvos work directory.

kyvosClusterName

Name of the Kyvos cluster to be deployed.

bundleAccessKey

Key to access Kyvos bundle.

bundleSecretKey

The secret key for Kyvos bundle.

sshPublicKey

Text of the SSH public key for authentication.

sshPrivateKey

Enter the SSH private Key text of the pem file.

NOTE: The text must be in base64 encoded.

kyvosLicenseFileValue

Enter a valid Kyvos license key.

NOTE: It should be base64 encoded

additionalLabels

Labels to be added to the resources. 

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

  • Change the value of the parameter kmCount to 0 in the kyvos-template.yaml file to go with wizard-based deployment.

  • Once created, you can validateif the resources meet the requirements for installing Kyvos on the Google cloud platform.

  • Post-deployment, for a non-SSH based cluster, if you use an existing Dataproc cluster and a new bucket for automated deployment on GCP, you must execute the  dataproc.sh  script on the master node of Dataproc after modifying the values of  DEPLOYMENT_BUCKET,  WORK_DIR,  COPY_LIB, and  DATAPROC_VERSION  to the name of the existing bucket. Then, sync the library and configuration files from the Kyvos Manager on the  Dataproc page. 

Deploying resources

To deploy your resources, use the gcloud command-line tool and execute the following using your configuration file:

gcloud deployment-manager deployments create <Name of resources> --config kyvos-template.yaml

If the deployment is successful, you will receive a message like the following example

Create operation operation-1432319707382-516afeb5d00f1-b864f0e7-b7103978 completed successfully.NAME TYPE STATE ERRORSquickstart-deployment compute.v1.instance COMPLETED -

Check your new deployment

To check the status of the deployment, run the following command

gcloud deployment-manager deployments describe <resource name>

...

qeCount

The number of instances to be used as query engines.

qeInstanceType

Instance type of query engine (n2-highmem-4).

qeDataVolumeSizeGB

Size of the disk to be attached with query engines.

qeCacheVolumeSizeGB

Size of the disk to be attached for the cache.

qeCacheVolumeCount

The number of disks to be attached to the cache.

qeCacheVolumeType

Type of the disk for cache storage (pd-ssd).

biCount

Enter the number of instances to be used as the BI server.

biInstanceType

Instance type of BI Server (n2-standard-8).

biBootScript

startup-script-bi.sh

sourceImage

Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-2024-2

RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

biVolumeCount

The number of disks to be attached to the BI Server.

biVolumeSizeGB

Size of the disk to be attached to the BI Server.

biVolumeType

Type of the disk for BI server (pd-ssd)

createGcpFunctions

Set the value as true to configure GCP Functions in Kyvos.

createDataProc

Enter true to create Dataproc. 

dataprocMetastoreURI

Enter the Metastore URI if you want to deploy Kyvos with no Spark configuration.

createGKE

Enter the value as True or False.

  • True: To create Kubernetes cluster.

gkeWorkerInitialNodeCount

Enter the initial worker node count for the Kubernetes cluster.

NOTE: The default value is 1.

gkeWorkerInstancetype

Enter the worker node instance type for the Kubernetes cluster.

NOTE: n2-standard-16 is the minimum configuration. Instance type smaller than this aren't supported.

minWorkerNodeCount

Enter the minimum worker node count.

maxWorkerNodeCount

Enter the maximum worker node count.

createDataProc

Enter the value as True or False.

  • True: If you want to deploy Kyvos with Spark Configuration.

enableComponentGateway

Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.

sharedDataprocCluster

Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.  

dataProcNetworkTags

Provide a list of comma-separated network tags to be added to the Dataproc cluster.

Example: dataProcNetworkTags : ["abc","xyz"]

enableSshFlag

Set the value to true to enable SSH to the Dataproc cluster.

enableLivy

Set the value of Livy to True if using Dataproc version 2.1.11-debian11.

masterInstanceCount

The number of master nodes. For example, 1 

masterInstanceType

Instance type of master node (n2-highmem-4)

masterInstanceVolumeType

Type of the disk for master node (pd-ssd)

workerInstanceCount

The number of worker instances.

workerInstanceType

Instance type of worker node (n2-highmem-8)

workerInstanceVolumeType

Type of the disk for worker node (pd-ssd)

enableDataProcMetastore

Set the value as true to allow external Dataproc metastore.
NOTE: Existing metastore is not supported if the value of the createVPC is true

dataProcMetastoreProjectId

If enable DataProcMetastore is set as true, provide the name of the metastore project ID.

dataProcMetaStoreName

Provide the name of the metastore name.

dataProcVersion

Supported version is 2.1.11-debian11

enableAutoScaling

Set the value as true to enable the autoscaling of cluster nodes.

existingAutoScalingPolicyName

Provide the name of the existing autoscaling property, if any.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMinInstanceCount

Specify the number of minimum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMaxInstanceCount

Specify the number of maximum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

existingDataprocClusterName

Enter the name of the existing Dataproc cluster. 

NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. 

sshPrivateKeyDataproc

The private key of existing Dataproc.

NOTE: The private key should be base64 encoded

dataprocUsername

Name of the user. 

createServiceAccount

 Change the value to false if you want to use the existing Service Account.

serviceAccountName

Enter the service account name to be attached to all Kyvos Virtual Machines. 

secretManagerName

Provide the name of the existing Secret Manager.

secretManagerProjectId

Provide the name of the Project ID in which the existing secret manager is created.

customPrefixNameDataproc

A prefix is to be added before the name of Dataproc.

customPrefixNameGKE

A prefix is to be added before GKE.

customPrefixNameBI

A prefix is to be added before the name of BI virtual machines.

customPrefixNameBIDisk

A prefix is to be added before the name of BI Disks.

customPrefixNameQE

A prefix is to be added before the name of Query Engine virtual machines.

customPrefixNameQEDisk

A prefix is to be added before the name of Query Engine disks.

customPrefixNameKM

A prefix is to be added before the name of the Kyvos Manager virtual machine. 

customPrefixNameKMDataDisk

A prefix is to be added before the name of the Kyvos Manager disk.

zone

The zone in which you want to deploy the resources.

bucketName

Enter the bucket name (if the bucket is not present, a new bucket would be created).

kyvosWorkDir

Provide the path of the Kyvos work directory.

kyvosClusterName

Name of the Kyvos cluster to be deployed.

bundleAccessKey

Key to access Kyvos bundle.

bundleSecretKey

The secret key for Kyvos bundle.

sshPublicKey

Text of the SSH public key for authentication.

sshPrivateKey

Enter the SSH private Key text of the pem file.

NOTE: The text must be in base64 encoded.

kyvosLicenseFileValue

Enter a valid Kyvos license key.

NOTE: It should be base64 encoded

additionalLabels

Labels to be added to the resources. 

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

Possible values in volume type fields:

For SSD type disk: pd-ssd

For standard disk: pd-standard

To create Dataproc for use with multiple build connections, use the multiple-build-cluster.tar file from the GCP Installation Files folder. In this case, use the dataproc-template.yaml file to create Dataproc.

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

All the build clusters must use the same GCP Dataproc metastore.

In the dataproc-template.yaml file, provide information as:

  1. For bucketName parameter, mention the bucket name used in the Kyvos deployment.

  2. For serviceAccountName parameter, mention the service account used in Kyvos resources.

  3. For kyvosWorkDir parameter, provide the working directory of Kyvos.

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

After filling all the required details in the dataproc-template.yaml file, run the following command:

gcloud deployment-manager deployments create <Name of resources> --config dataproc-template.yaml

Aura tab
summaryKyvos resources with Kubernetes and Dataproc
paramsJTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMHdpdGglMjBLdWJlcm5ldGVzJTIwYW5kJTIwRGF0YXByb2MlMjIlN0Q=

Parameter

Description

createVPC

Set the value as true to create a new VPC.

customPrefixNameVPC

A custom prefix name for a VPC. 

customPrefixNameSubNetwork

A custom prefix name for the subnetwork. 

ipCidrRange

Enter the VPC CIDR range in which VPC will be created. 

vpc

Enter the VPC within which you want to deploy the resources.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

vpcProjectId

Enter the Project ID of your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

subnetwork

Enter the subnetwork within your VPC.

NOTE: You must leave this field blank if the parameter createVPC is set to true. 

createNetworkFirewall

To create firewall rules, set the value of this parameter to false.

NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally. 

gkeSubnetName

Enter the name of an existing Subnet in which you want to deploy GKE Cluster. If left blank subnetwork name will be used.

secondaryRangeName1

Enter the Secondary IPv4 ranges name for GKE Cluster creation.

NOTE: This must be preconfigured if using an existing VPC.
The range should have a minimum masking of /22.

For more information, see Google documentation.

secondaryRangeName2

Enter the Secondary IPv4 ranges name for GKE Cluster creation.

NOTE: This must be preconfigured if using an existing VPC.
The range should have a minimum masking of /22.

For more information, see Google documentation.

enableVpcConnector

Keep the value of this parameter as false if the value of the parameter (createVPC) is true. 

vpcConnectorName

Enter the name of the VPC Connector to be used with GCP functions. 

kmCount

The number of Kyvos Manager instances to be launched.

kmInstanceType

Instance type of Kyvos Manager (n2-standard-4). 

kmVolumeSizeGB

Size of the disk to be attached to the Kyvos Manager.

kmBootScript startup-script

startup-script-km.sh

kmSourceImage

Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-2024-2

RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

webServerPublicIP

Change the value to true if public IP for the Web Server is needed.
NOTE: Change the value to true if the value of createVPC is true

kmVolumeType

Type of the disk for KM (pd-ssd).

hostNameBasedDeployment

Change the value to true to use the hostname for the cluster deployment.

qeCount

The number of instances to be used as query engines.

qeInstanceType

Instance type of query engine (n2-highmem-4).

qeDataVolumeSizeGB

Size of the disk to be attached with query engines.

qeCacheVolumeSizeGB

Size of the disk to be attached for the cache.

qeCacheVolumeCount

The number of disks to be attached to the cache.

qeCacheVolumeType

Type of the disk for cache storage (pd-ssd).

biCount

Enter the number of instances to be used as the BI server.

biInstanceType

Instance type of BI Server (n2-standard-8).

biBootScript

startup-script-bi.sh

sourceImage

Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-2024-2

RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5

biVolumeCount

The number of disks to be attached to the BI Server.

biVolumeSizeGB

Size of the disk to be attached to the BI Server.

biVolumeType

Type of the disk for BI server (pd-ssd)

createGcpFunctions

Set the value as true to configure GCP Functions in Kyvos.

createDataProc

Enter true to create Dataproc. 

dataprocMetastoreURI

Enter the Metastore URI if you want to deploy Kyvos with no Spark configuration.

createGKE

Enter the value as True or False.

  • True: To create Kubernetes cluster.

gkeWorkerInitialNodeCount

Enter the initial worker node count for the Kubernetes cluster.

NOTE: The default value is 1.

gkeWorkerInstancetype

Enter the worker node instance type for the Kubernetes cluster.

NOTE: n2-standard-16 is the minimum configuration. Instance type smaller than this aren't supported.

minWorkerNodeCount

Enter the minimum worker node count.

maxWorkerNodeCount

Enter the maximum worker node count.

createDataProc

Enter the value as True or False.

  • True: If you want to deploy Kyvos with Spark Configuration.

enableComponentGateway

Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.

sharedDataprocCluster

Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.  

dataProcNetworkTags

Provide a list of comma-separated network tags to be added to the Dataproc cluster.

Example: dataProcNetworkTags : ["abc","xyz"]

enableSshFlag

Set the value to true to enable SSH to the Dataproc cluster.

enableLivy

Set the value of Livy to True if using Dataproc version 2.1.11-debian11.

masterInstanceCount

The number of master nodes. For example, 1 

masterInstanceType

Instance type of master node (n2-highmem-4)

masterInstanceVolumeType

Type of the disk for master node (pd-ssd)

workerInstanceCount

The number of worker instances.

workerInstanceType

Instance type of worker node (n2-highmem-8)

workerInstanceVolumeType

Type of the disk for worker node (pd-ssd)

enableDataProcMetastore

Set the value as true to allow external Dataproc metastore.
NOTE: Existing metastore is not supported if the value of the createVPC is true

dataProcMetastoreProjectId

If enable DataProcMetastore is set as true, provide the name of the metastore project ID.

dataProcMetaStoreName

Provide the name of the metastore name.

dataProcVersion

Supported version is 2.1.11-debian11

enableAutoScaling

Set the value as true to enable the autoscaling of cluster nodes.

existingAutoScalingPolicyName

Provide the name of the existing autoscaling property, if any.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMinInstanceCount

Specify the number of minimum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

secondaryWorkerMaxInstanceCount

Specify the number of maximum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.

existingDataprocClusterName

Enter the name of the existing Dataproc cluster. 

NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. 

sshPrivateKeyDataproc

The private key of existing Dataproc.

NOTE: The private key should be base64 encoded

dataprocUsername

Name of the user. 

createServiceAccount

 Change the value to false if you want to use the existing Service Account.

serviceAccountName

Enter the service account name to be attached to all Kyvos Virtual Machines. 

secretManagerName

Provide the name of the existing Secret Manager.

secretManagerProjectId

Provide the name of the Project ID in which the existing secret manager is created.

customPrefixNameDataproc

A prefix is to be added before the name of Dataproc.

customPrefixNameGKE

A prefix is to be added before GKE.

customPrefixNameBI

A prefix is to be added before the name of BI virtual machines.

customPrefixNameBIDisk

A prefix is to be added before the name of BI Disks.

customPrefixNameQE

A prefix is to be added before the name of Query Engine virtual machines.

customPrefixNameQEDisk

A prefix is to be added before the name of Query Engine disks.

customPrefixNameKM

A prefix is to be added before the name of the Kyvos Manager virtual machine. 

customPrefixNameKMDataDisk

A prefix is to be added before the name of the Kyvos Manager disk.

zone

The zone in which you want to deploy the resources.

bucketName

Enter the bucket name (if the bucket is not present, a new bucket would be created).

kyvosWorkDir

Provide the path of the Kyvos work directory.

kyvosClusterName

Name of the Kyvos cluster to be deployed.

bundleAccessKey

Key to access Kyvos bundle.

bundleSecretKey

The secret key for Kyvos bundle.

sshPublicKey

Text of the SSH public key for authentication.

sshPrivateKey

Enter the SSH private Key text of the pem file.

NOTE: The text must be in base64 encoded.

kyvosLicenseFileValue

Enter a valid Kyvos license key.

NOTE: It should be base64 encoded

additionalLabels

Labels to be added to the resources. 

Tip

Tip

If you encounter any issue while processing the semantic model through Kubernetes, you cab debug the issue. For more details about debugging, see the Kubernetes documentation.

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

  • Change the value of the parameter kmCount to 0 in the kyvos-template.yaml file to go with wizard-based deployment.

  • Once created, you can validateif the resources meet the requirements for installing Kyvos on the Google cloud platform.

  • Post-deployment, for a non-SSH based cluster, if you use an existing Dataproc cluster and a new bucket for automated deployment on GCP, you must execute the  dataproc.sh  script on the master node of Dataproc after modifying the values of  DEPLOYMENT_BUCKET,  WORK_DIR,  COPY_LIB, and  DATAPROC_VERSION  to the name of the existing bucket. Then, sync the library and configuration files from the Kyvos Manager on the  Dataproc page. 

Deploying resources

To deploy your resources, use the gcloud command-line tool and execute the kyvos-deployment.sh file (provided in GCP installation files)

If the deployment is successful, you will receive a message like the following example

Create operation operation-1432319707382-516afeb5d00f1-b864f0e7-b7103978 completed successfully.NAME TYPE STATE ERRORSquickstart-deployment compute.v1.instance COMPLETED -

whereas,

NAME:<resource_name>
TYPE: compute.v1.disk
STATE: COMPLETED

Check your new deployment

To check the status of the deployment, run the following command

Code Block
gcloud deployment-manager deployments describe <resource name>

A new bucket will be created for the Dataproc cluster.

Using existing Service Account

Once Kyvos resources are created using Kubernetes, execute the following commands using the gcloud CLI to link the Kubernetes Service account to the IAM Service account.

gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:PROJECT_ID.svc.id.goog[kyvos-monitoring/default]"

gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:PROJECT_ID.svc.id.goog[kyvos-compute/default]"

In the above-mentioned commands, replace the following:

  • IAM_SA_NAME: The name of your new IAM service account.

  • IAM_SA_PROJECT_ID: The project ID of your IAM service account.

  • PROJECT_ID: The project ID of your Google Cloud.

Panel
panelIconIdatlassian-note
panelIcon:note:
bgColor#DEEBFF

Note

After creating resources using scripts, the configuration of the Kyvos GCP cluster on Kyvos Manager is no longer needed. Once the resources are successfully created, deployment of the Kyvos cluster will commence automatically on logging into the Kyvos Manager portal.

...