Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace
Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)
...
Before you begin
In addition to the prerequisites, please ensure the following settings are enabled on your GCP project.
...
Creating resources using script
Anchor |
---|
| creatingscript |
---|
| creatingscript |
---|
|
...
params | JTdCJTIyZ2VuZXJhbCUyMiUzQSU3QiUyMnRhYlNwYWNpbmclMjIlM0EwJTJDJTIydGFiV2lkdGglMjIlM0ExMDAlMkMlMjJ0YWJIZWlnaHQlMjIlM0E1MCUyQyUyMmRpcmVjdGlvbiUyMiUzQSUyMmhvcml6b250YWwlMjIlN0QlMkMlMjJjb250ZW50JTIyJTNBJTdCJTIyYmFja2dyb3VuZENvbG9yJTIyJTNBJTdCJTIyY29sb3IlMjIlM0ElMjIlMjNmZmYlMjIlN0QlMkMlMjJib3JkZXIlMjIlM0ElN0IlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyd2lkdGglMjIlM0ExJTJDJTIydG9wJTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmNvbG9yJTIyJTNBJTdCJTIybGlnaHQlMjIlM0ElMjIlMjNjY2NlY2YlMjIlN0QlN0QlMkMlMjJwYWRkaW5nJTIyJTNBJTdCJTIydG9wJTIyJTNBMTAlMkMlMjJyaWdodCUyMiUzQTEwJTJDJTIyYm90dG9tJTIyJTNBMTAlMkMlMjJsZWZ0JTIyJTNBMTAlN0QlN0QlMkMlMjJhY3RpdmUlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzZjU4MjI3JTIyJTdEJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzMDAwMDAwJTIyJTdEJTJDJTIydGV4dEFsaWduJTIyJTNBJTIybGVmdCUyMiUyQyUyMmZvbnRXZWlnaHQlMjIlM0ElMjJib2xkJTIyJTdEJTdEJTJDJTIyaG92ZXIlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSUyMiUyM2RmZTFlNiUyMiU3RCUyQyUyMnRleHQlMjIlM0ElN0IlMjJmb250U2l6ZSUyMiUzQTE4JTJDJTIyY29sb3IlMjIlM0ElMjIlMjM1ZTZjODQlMjIlMkMlMjJ0ZXh0QWxpZ24lMjIlM0ElMjJsZWZ0JTIyJTJDJTIyZm9udFdlaWdodCUyMiUzQSUyMmxpZ2h0ZXIlMjIlN0QlN0QlMkMlMjJpbmFjdGl2ZSUyMiUzQSU3QiUyMmJhY2tncm91bmRDb2xvciUyMiUzQSU3QiUyMmNvbG9yJTIyJTNBJTIyJTIzZjRmNWY3JTIyJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSUyMiUyMzVlNmM4NCUyMiUyQyUyMnRleHRBbGlnbiUyMiUzQSUyMmxlZnQlMjIlMkMlMjJmb250V2VpZ2h0JTIyJTNBJTIybGlnaHRlciUyMiU3RCUyQyUyMmJvcmRlciUyMiUzQSU3QiUyMnRvcCUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJ3aWR0aCUyMiUzQTElMkMlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyY29sb3IlMjIlM0ElN0IlMjJsaWdodCUyMiUzQSUyMiUyM2NjY2VjZiUyMiU3RCU3RCU3RCU3RA== |
---|
Aura tab |
---|
summary | Kyvos resources and Dataproc |
---|
params | JTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMGFuZCUyMERhdGFwcm9jJTIyJTdE |
---|
|
...
Download the gcp.tar file from the GCP Installation Files folder on your workstation.
On your workstation, install the gcloud command-line tool.
Configure the gcloud command-line tool to use your project using the following command.
gcloud config set project [MY_PROJECT]
Here, replace [MY_PROJECT] with your project ID.
Copy the script tar file and untar it.
Update the following parameters in the kyvos-template.yaml (provided in the gcp.tar) as per your business requirement.
...
Parameter
...
Description
...
createVPC
...
Set the value as true to create a new VPC.
...
customPrefixNameVPC
...
A custom prefix name for a VPC.
...
customPrefixNameSubNetwork
...
A custom prefix name for the subnetwork.
...
ipCidrRange
...
Enter the VPC CIDR range in which VPC will be created.
...
vpc
...
Enter the VPC within which you want to deploy the resources.
NOTE: You must leave this field blank if the parameter createVPC is set to true.
...
vpcProjectId
...
Enter the Project ID of your VPC.
NOTE: You must leave this field blank if the parameter createVPC is set to true.
...
createNetworkFirewall
...
To create firewall rules, set the value of this parameter to true.
NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally.
...
subnetwork
...
Enter the subnetwork within your VPC.
NOTE: You must leave this field blank if the parameter createVPC is set to true.
...
enableVpcConnector
...
Keep the value of this parameter as false if the value of the parameter (createVPC) is true.
...
vpcConnectorName
...
Enter the name of the VPC Connector to be used with GCP functions.
...
kmCount
...
The number of Kyvos Manager instances to be launched.
...
kmInstanceType
...
Instance type of Kyvos Manager (n2-standard-4).
...
kmVolumeSizeGB
...
Size of the disk to be attached to the Kyvos Manager.
...
kmBootScript startup-script
...
startup-script-km.sh
...
kmSourceImage
...
Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07
RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5
...
webServerPublicIP
...
Change the value to true if public IP for the Web Server is needed.
NOTE: Change the value to true if the value of createVPC is true
...
kmVolumeType
...
Type of the disk for KM (pd-ssd).
...
hostNameBasedDeployment
...
Change the value to true to use the hostname for the cluster deployment.
...
qeCount
...
The number of instances to be used as query engines.
...
qeInstanceType
...
Instance type of query engine (n2-highmem-4).
...
qeDataVolumeSizeGB
...
Size of the disk to be attached with query engines.
...
qeCacheVolumeSizeGB
...
Size of the disk to be attached for the cache.
...
qeCacheVolumeCount
...
The number of disks to be attached to the cache.
...
qeCacheVolumeType
...
Type of the disk for cache storage (pd-ssd).
...
biCount
...
Enter the number of instances to be used as the BI server.
...
biInstanceType
...
Instance type of BI Server (n2-standard-8).
...
biBootScript
...
startup-script-bi.sh
...
sourceImage
...
Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07
RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5
...
biVolumeCount
...
The number of disks to be attached to the BI Server.
...
biVolumeSizeGB
...
Size of the disk to be attached to the BI Server.
...
biVolumeType
...
Type of the disk for BI server (pd-ssd)
...
createGcpFunctions
...
Set the value as true to configure GCP Functions in Kyvos.
...
createDataProc
...
Enter true to create Dataproc.
...
enableComponentGateway
...
Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.
...
sharedDataprocCluster
...
Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.
...
dataProcNetworkTags
...
Provide a list of comma-separated network tags to be added to the Dataproc cluster.
Example: dataProcNetworkTags : ["abc","xyz"]
...
enableSshFlag
...
Set the value to true to enable SSH to the Dataproc cluster.
...
enableLivy
...
Set the value of Livy to True if using Dataproc version 2.1.11-debian11.
...
masterInstanceCount
...
The number of master nodes. For example, 1
...
masterInstanceType
...
Instance type of master node (n2-highmem-4)
...
masterInstanceVolumeType
...
Type of the disk for master node (pd-ssd)
...
workerInstanceCount
...
The number of worker instances.
...
workerInstanceType
...
Instance type of worker node (n2-highmem-8)
...
workerInstanceVolumeType
...
Type of the disk for worker node (pd-ssd)
...
enableDataProcMetastore
...
Set the value as true to allow external Dataproc metastore.
NOTE: Existing metastore is not supported if the value of the createVPC is true
...
dataProcMetastoreProjectId
...
If enable DataProcMetastore is set as true, provide the name of the metastore project ID.
...
dataProcMetaStoreName
...
Provide the name of the metastore name.
...
dataProcVersion
...
Supported version is 2.1.11-debian11
...
enableAutoScaling
...
Set the value as true to enable the autoscaling of cluster nodes.
...
existingAutoScalingPolicyName
...
Provide the name of the existing autoscaling property, if any.
NOTE: Use this configuration only if enableAutoScaling is set as true.
...
secondaryWorkerMinInstanceCount
...
Specify the number of minimum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.
...
secondaryWorkerMaxInstanceCount
...
Specify the number of maximum worker instances to be kept running while scaling.
NOTE: Use this configuration only if enableAutoScaling is set as true.
...
existingDataprocClusterName
...
Enter the name of the existing Dataproc cluster.
NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false.
...
sshPrivateKeyDataproc
...
The private key of existing Dataproc.
NOTE: The private key should be base64 encoded
...
dataprocUsername
...
Name of the user.
...
createServiceAccount
...
Change the value to false if you want to use the existing Service Account.
...
serviceAccountName
...
Enter the service account name to be attached to all Kyvos Virtual Machines.
...
secretManagerName
...
Provide the name of the existing Secret Manager.
...
secretManagerProjectId
...
Provide the name of the Project ID in which the existing secret manager is created.
...
customPrefixNameDataproc
...
A prefix is to be added before the name of Dataproc.
...
customPrefixNameBI
...
A prefix is to be added before the name of BI virtual machines.
...
customPrefixNameBIDisk
...
A prefix is to be added before the name of BI Disks.
...
customPrefixNameQE
...
A prefix is to be added before the name of Query Engine virtual machines.
...
customPrefixNameQEDisk
...
A prefix is to be added before the name of Query Engine disks.
...
customPrefixNameKM
...
A prefix is to be added before the name of the Kyvos Manager virtual machine.
...
customPrefixNameKMDataDisk
...
A prefix is to be added before the name of the Kyvos Manager disk.
...
zone
...
The zone in which you want to deploy the resources.
...
bucketName
...
Enter the bucket name (if the bucket is not present, a new bucket would be created).
...
kyvosWorkDir
...
Provide the path of the Kyvos work directory.
...
kyvosClusterName
...
Name of the Kyvos cluster to be deployed.
...
bundleAccessKey
...
Key to access Kyvos bundle.
...
bundleSecretKey
...
The secret key for Kyvos bundle.
...
sshPublicKey
...
Text of the SSH public key for authentication.
...
sshPrivateKey
...
Enter the SSH private Key text of the pem file.
NOTE: The text must be in base64 encoded.
...
kyvosLicenseFileValue
...
Enter a valid Kyvos license key.
NOTE: It should be base64 encoded
...
additionalLabels
...
Labels to be added to the resources.
Panel |
---|
panelIconId | atlassian-note |
---|
panelIcon | :note: |
---|
bgColor | #DEEBFF |
---|
|
Note Possible values in volume type fields: For SSD type disk: pd-ssd For standard disk: pd-standard |
Aura tab |
---|
summary | Kyvos resources and Dataproc for Multiple build connections |
---|
params | JTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMGFuZCUyMERhdGFwcm9jJTIwZm9yJTIwTXVsdGlwbGUlMjBidWlsZCUyMGNvbm5lY3Rpb25zJTIyJTdE |
---|
|
Download the gcp.tar file from the GCP Installation Files folder on your workstation.
On your workstation, install the gcloud command-line tool.
Configure the gcloud command-line tool to use your project using the following command.
gcloud config set project [MY_PROJECT]
Here, replace [MY_PROJECT] with your project ID.
...
Copy the script tar file and untar it.
Enter details as:
Aura tab collection |
---|
params | JTdCJTIyZ2VuZXJhbCUyMiUzQSU3QiUyMnRhYlNwYWNpbmclMjIlM0EwJTJDJTIydGFiV2lkdGglMjIlM0ExMDAlMkMlMjJ0YWJIZWlnaHQlMjIlM0E1MCUyQyUyMmRpcmVjdGlvbiUyMiUzQSUyMmhvcml6b250YWwlMjIlN0QlMkMlMjJjb250ZW50JTIyJTNBJTdCJTIyYmFja2dyb3VuZENvbG9yJTIyJTNBJTdCJTIyY29sb3IlMjIlM0ElMjIlMjNmZmYlMjIlN0QlMkMlMjJib3JkZXIlMjIlM0ElN0IlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyd2lkdGglMjIlM0ExJTJDJTIydG9wJTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmNvbG9yJTIyJTNBJTdCJTIybGlnaHQlMjIlM0ElMjIlMjNjY2NlY2YlMjIlN0QlN0QlMkMlMjJwYWRkaW5nJTIyJTNBJTdCJTIydG9wJTIyJTNBMTAlMkMlMjJyaWdodCUyMiUzQTEwJTJDJTIyYm90dG9tJTIyJTNBMTAlMkMlMjJsZWZ0JTIyJTNBMTAlN0QlN0QlMkMlMjJhY3RpdmUlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzZjU4MjI3JTIyJTdEJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSU3QiUyMmxpZ2h0JTIyJTNBJTIyJTIzMDAwMDAwJTIyJTdEJTJDJTIydGV4dEFsaWduJTIyJTNBJTIybGVmdCUyMiUyQyUyMmZvbnRXZWlnaHQlMjIlM0ElMjJib2xkJTIyJTdEJTdEJTJDJTIyaG92ZXIlMjIlM0ElN0IlMjJiYWNrZ3JvdW5kQ29sb3IlMjIlM0ElN0IlMjJjb2xvciUyMiUzQSUyMiUyM2RmZTFlNiUyMiU3RCUyQyUyMnRleHQlMjIlM0ElN0IlMjJmb250U2l6ZSUyMiUzQTE4JTJDJTIyY29sb3IlMjIlM0ElMjIlMjM1ZTZjODQlMjIlMkMlMjJ0ZXh0QWxpZ24lMjIlM0ElMjJsZWZ0JTIyJTJDJTIyZm9udFdlaWdodCUyMiUzQSUyMmxpZ2h0ZXIlMjIlN0QlN0QlMkMlMjJpbmFjdGl2ZSUyMiUzQSU3QiUyMmJhY2tncm91bmRDb2xvciUyMiUzQSU3QiUyMmNvbG9yJTIyJTNBJTIyJTIzZjRmNWY3JTIyJTdEJTJDJTIydGV4dCUyMiUzQSU3QiUyMmZvbnRTaXplJTIyJTNBMTYlMkMlMjJjb2xvciUyMiUzQSUyMiUyMzVlNmM4NCUyMiUyQyUyMnRleHRBbGlnbiUyMiUzQSUyMmxlZnQlMjIlMkMlMjJmb250V2VpZ2h0JTIyJTNBJTIybGlnaHRlciUyMiU3RCUyQyUyMmJvcmRlciUyMiUzQSU3QiUyMnRvcCUyMiUzQXRydWUlMkMlMjJsZWZ0JTIyJTNBdHJ1ZSUyQyUyMnJpZ2h0JTIyJTNBdHJ1ZSUyQyUyMmJvdHRvbSUyMiUzQXRydWUlMkMlMjJ3aWR0aCUyMiUzQTElMkMlMjJzdHlsZSUyMiUzQSUyMnNvbGlkJTIyJTJDJTIyY29sb3IlMjIlM0ElN0IlMjJsaWdodCUyMiUzQSUyMiUyM2NjY2VjZiUyMiU3RCU3RCU3RCU3RA== |
---|
|
Aura tab |
---|
summary | Kyvos resources and Dataproc for Multiple build connections |
---|
params | JTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMGFuZCUyMERhdGFwcm9jJTIwZm9yJTIwTXVsdGlwbGUlMjBidWlsZCUyMGNvbm5lY3Rpb25zJTIyJTdE |
---|
|
Parameter | Description |
---|
createVPC | Set the value as true to create a new VPC. | customPrefixNameVPC | A custom prefix name for a VPC. | customPrefixNameSubNetwork | A custom prefix name for the subnetwork. | ipCidrRange | Enter the VPC CIDR range in which VPC will be created. | vpc | Enter the VPC within which you want to deploy the resources. NOTE: You must leave this field blank if the parameter createVPC is set to true. | vpcProjectId | Enter the Project ID of your VPC. NOTE: You must leave this field blank if the parameter createVPC is set to true. | createNetworkFirewall | To create firewall rules, set the value of this parameter to true. NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally. | subnetwork | Enter the subnetwork within your VPC. NOTE: You must leave this field blank if the parameter createVPC is set to true. | enableVpcConnector | Keep the value of this parameter as false if the value of the parameter (createVPC) is true. | vpcConnectorName | Enter the name of the VPC Connector to be used with GCP functions. | kmCount | The number of Kyvos Manager instances to be launched. | kmInstanceType | Instance type of Kyvos Manager (n2-standard-4). | kmVolumeSizeGB | Size of the disk to be attached to the Kyvos Manager. | kmBootScript startup-script | startup-script-km.sh | kmSourceImage | Enter the following path: Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07 RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5 | webServerPublicIP | Change the value to true if public IP for the Web Server is needed. NOTE: Change the value to true if the value of createVPC is true | kmVolumeType | Type of the disk for KM (pd-ssd). | hostNameBasedDeployment | Change the value to true to use the hostname for the cluster deployment | qeCount | The number of instances to be used as query engines. | qeInstanceType | Instance type of query engine (n2-highmem-4). | qeDataVolumeSizeGB | Size of the disk to be attached with query engines. | qeCacheVolumeSizeGB | Size of the disk to be attached for the cache. | qeCacheVolumeCount | The number of disks to be attached to the cache. | qeCacheVolumeType | Type of the disk for cache storage (pd-ssd). | biCount | Enter the number of instances to be used as the BI server. | biInstanceType | Instance type of BI Server (n2-standard-8). | biBootScript | startup-script-bi.sh | sourceImage | Enter the following path: Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07 RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5 | biVolumeCount | The number of disks to be attached to the BI Server. | biVolumeSizeGB | Size of the disk to be attached to the BI Server. | biVolumeType | Type of the disk for BI server (pd-ssd) | createGcpFunctions | Set the value as true to configure GCP Functions in Kyvos. | createDataProc | Enter true or false as per your requirement. | enableComponentGateway | Set enableComponentGateway to true to get publicly accessible URL for Dataproc. | sharedDataprocCluster | Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster. Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop. | dataProcNetworkTags | Provide a list of comma-separated network tags to be added on Dataproc cluster. Example: dataProcNetworkTags : ["abc","xyz"] | enableSshFlag | Set the value to true to enable SSH to the Dataproc cluster. | enableLivy | Set the value of Livy to True if using Dataproc version 2.1.11-debian11. | masterInstanceCount | The number of master nodes. For example, 1 | masterInstanceType | Instance type of master node (n2-highmem-4) | masterInstanceVolumeType | Type of the disk for master node (pd-ssd) | workerInstanceCount | The number of worker instances. | workerInstanceType | Instance type of worker node (n2-highmem-8) | workerInstanceVolumeType | Type of the disk for worker node (pd-ssd is recommended) | enableDataProcMetastore | Set the value as true to allow external Dataproc metastore. NOTE: Existing metastore is not supported if the value of the createVPC is true | dataProcMetastoreProjectId | If enable DataProcMetastore is set as true, provide the name of the metastore project ID. | dataProcMetaStoreName | Provide the name of the metastore name. | dataProcVersion | Supported version: | enableAutoScaling | Set the value as true to enable the autoscaling of cluster nodes. | existingAutoScalingPolicyName | Provide the name of the existing autoscaling property, if any. NOTE: Use this configuration only if enableAutoScaling is set as true. | secondaryWorkerMinInstanceCount | Specify the number of minimum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true. | secondaryWorkerMaxInstanceCount | Specify the number of maximum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true. | existingDataprocClusterName | Enter the name of the existing Dataproc cluster. NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. | sshPrivateKeyDataproc | The private key of existing Dataproc. NOTE: The private key should be base64 encoded | dataprocUsername | Name of the existing user. | createServiceAccount | Change the value to false if you want to use the existing Service Account. | serviceAccountName | Enter the service account name to be attached to all Kyvos Virtual Machines. | secretManagerName | Provide the name of the existing Secret Manager. | secretManagerProjectId | Provide the name of the Project ID in which the existing secret manager is created. | customPrefixNameDataproc | A prefix is to be added before the name of Dataproc. | customPrefixNameBI | A prefix is to be added before the name of BI virtual machines. | customPrefixNameBIDisk | A prefix is to be added before the name of BI Disks. | customPrefixNameQE | A prefix is to be added before the name of Query Engine virtual machines. | customPrefixNameQEDisk | A prefix is to be added before the name of Query Engine disks. | customPrefixNameKM | A prefix is to be added before the name of the Kyvos Manager virtual machine. | customPrefixNameKMDataDisk | A prefix is to be added before the name of the Kyvos Manager disk. | zone | The zone in which you want to deploy the resources. | bucketName | Enter the bucket name (if the bucket is not present, a new bucket would be created). | kyvosWorkDir | Provide the path of the Kyvos work directory. | kyvosClusterName | Name of the Kyvos cluster to be deployed. | bundleAccessKey | Key to access Kyvos bundle. | bundleSecretKey | The secret key for Kyvos bundle. | sshPublicKey | Text of the SSH public key for authentication. | sshPrivateKey | Enter the SSH private Key text of the pem file. NOTE: The text must be in base64 encoded. | kyvosLicenseFileValue | Enter a valid Kyvos license key. NOTE: It should be base64 encoded | additionalLabels | Labels to be added to the resources. |
Panel |
---|
panelIconId | atlassian-note |
---|
panelIcon | :note: |
---|
bgColor | #DEEBFF |
---|
| Note Possible values in volume type fields: For SSD type disk: pd-ssd For standard disk: pd-standard |
To create Dataproc for use with multiple build connections, use the multiple-build-cluster.tar file from the GCP Installation Files folder. In this case, use the dataproc-template.yaml file to create Dataproc. Panel |
---|
panelIconId | atlassian-note |
---|
panelIcon | :note: |
---|
bgColor | #DEEBFF |
---|
| Note All the build clusters must use the same GCP Dataproc metastore. |
In the dataproc-template.yaml file, provide information as: For bucketName parameter, mention the bucket name used in the Kyvos deployment. For serviceAccountName parameter, mention the service account used in Kyvos resources. For kyvosWorkDir parameter, provide the working directory of Kyvos.
Panel |
---|
panelIconId | atlassian-note |
---|
panelIcon | :note: |
---|
bgColor | #DEEBFF |
---|
| Note After filling all the required details in the dataproc-template.yaml file, run the following command: gcloud deployment-manager deployments create <Name of resources> --config dataproc-template.yaml |
Aura tab |
---|
summary | Kyvos resources with Kubernetes and Dataproc |
---|
params | JTdCJTIydGl0bGUlMjIlM0ElMjJLeXZvcyUyMHJlc291cmNlcyUyMHdpdGglMjBLdWJlcm5ldGVzJTIwYW5kJTIwRGF0YXByb2MlMjIlN0Q= |
---|
|
Navigate to the kyvos-template.yaml file located in the GCP Installation Files and update the following parameters. Parameter | Description |
---|
createVPC | Set the value as true to create a new VPC. | customPrefixNameVPC | A custom prefix name for a VPC. | customPrefixNameSubNetwork | A custom prefix name for the subnetwork. | ipCidrRange | Enter the VPC CIDR range in which VPC will be created. | vpc | Enter the VPC within which you want to deploy the resources. NOTE: You must leave this field blank if the parameter createVPC is set to true. | vpcProjectId | Enter the Project ID of your VPC. NOTE: You must leave this field blank if the parameter createVPC is set to true. | subnetwork | Enter the subnetwork within your VPC. NOTE: You must leave this field blank if the parameter createVPC is set to true. | createNetworkFirewall | To create firewall rules, set the value of this parameter to false. NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally. | gkeSubnetName | Enter the name of an existing Subnet in which you want to deploy GKE Cluster. If left blank subnetwork name will be used. | secondaryRangeName1 | Enter the Secondary IPv4 ranges name for GKE Cluster creation. | secondaryRangeName2 | Enter the Secondary IPv4 ranges name for GKE Cluster creation. NOTE: This must be preconfigured if using an existing VPC. The range should havea minimum masking of /22. | enableVpcConnector | Keep the value of this parameter as false if the value of the parameter (createVPC) is true. | vpcConnectorName | Enter the name of the VPC Connector to be used with GCP functions. | kmCount | The number of Kyvos Manager instances to be launched. | kmInstanceType | Instance type of Kyvos Manager (n2-standard-4). | kmVolumeSizeGB | Size of the disk to be attached to the Kyvos Manager. | kmBootScript startup-script | startup-script-km.sh | kmSourceImage | Ubuntu- projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07 RHEL- projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5 | webServerPublicIP | Change the value to true if public IP for the Web Server is needed. NOTE: Change the value to true if the value of createVPC is true | kmVolumeType | Type of the disk for KM (pd-ssd). | hostNameBasedDeployment | Change the value to true to use the hostname for the cluster deployment. | qeCount | The number of instances to be used as query engines. | qeInstanceType | Instance type of query engine (n2-highmem-4). | qeDataVolumeSizeGB | Size of the disk to be attached with query engines. | qeCacheVolumeSizeGB | Size of the disk to be attached for the cache. | qeCacheVolumeCount | The number of disks to be attached to the cache. | qeCacheVolumeType | Type of the disk for cache storage (pd-ssd). | biCount | Enter the number of instances to be used as the BI server. | biInstanceType | Instance type of BI Server (n2-standard-8). | biBootScript | startup-script-bi.sh | sourceImage | Ubuntu: projects/dev-enterprises-273509/global/images/kyvos-ubuntu20-image-2023-11-07 RHEL: projects/dev-enterprises-273509/global/images/kyvos-rhel-image-2023-5 | biVolumeCount | The number of disks to be attached to the BI Server. | biVolumeSizeGB | Size of the disk to be attached to the BI Server. | biVolumeType | Type of the disk for BI server (pd-ssd) | createGcpFunctions | Set the value as true to configure GCP Functions in Kyvos. | createDataProc | Enter true to create Dataproc. | dataprocMetastoreURI | Enter the Metastore URI if you want to deploy Kyvos with no Spark configuration. | createGKE | Enter the value as True or False. | gkeWorkerInitialNodeCount | Enter the initial worker node count for the Kubernetes cluster. NOTE: The default value is 1. | gkeWorkerInstancetype | Enter the worker node instance type for the Kubernetes cluster. NOTE: n2-highmem-8 is the minimum configuration. Instance type smaller than this aren't supported. | minWorkerNodeCount | Enter the minimum worker node count. | maxWorkerNodeCount | Enter the maximum worker node count. | createDataProc | Enter the value as True or False. | enableComponentGateway | Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc. | sharedDataprocCluster | Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop. | dataProcNetworkTags | Provide a list of comma-separated network tags to be added to the Dataproc cluster. Example: dataProcNetworkTags : ["abc","xyz"] | enableSshFlag | Set the value to true to enable SSH to the Dataproc cluster. | enableLivy | Set the value of Livy to True if using Dataproc version 2.1.11-debian11. | masterInstanceCount | The number of master nodes. For example, 1 | masterInstanceType | Instance type of master node (n2-highmem-4) | masterInstanceVolumeType | Type of the disk for master node (pd-ssd) | workerInstanceCount | The number of worker instances. | workerInstanceType | Instance type of worker node (n2-highmem-8) | workerInstanceVolumeType | Type of the disk for worker node (pd-ssd) | enableDataProcMetastore | Set the value as true to allow external Dataproc metastore. NOTE: Existing metastore is not supported if the value of the createVPC is true | dataProcMetastoreProjectId | If enable DataProcMetastore is set as true, provide the name of the metastore project ID. | dataProcMetaStoreName | Provide the name of the metastore name. | dataProcVersion | Supported version is 2.1.11-debian11 | enableAutoScaling | Set the value as true to enable the autoscaling of cluster nodes. | existingAutoScalingPolicyName | Provide the name of the existing autoscaling property, if any. NOTE: Use this configuration only if enableAutoScaling is set as true. | secondaryWorkerMinInstanceCount | Specify the number of minimum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true. | secondaryWorkerMaxInstanceCount | Specify the number of maximum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true. | existingDataprocClusterName | Enter the name of the existing Dataproc cluster. NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false. | sshPrivateKeyDataproc | The private key of existing Dataproc. NOTE: The private key should be base64 encoded | dataprocUsername | Name of the user. | createServiceAccount | Change the value to false if you want to use the existing Service Account. | serviceAccountName | Enter the service account name to be attached to all Kyvos Virtual Machines. | secretManagerName | Provide the name of the existing Secret Manager. | secretManagerProjectId | Provide the name of the Project ID in which the existing secret manager is created. | customPrefixNameDataproc | A prefix is to be added before the name of Dataproc. | customPrefixNameBI | A prefix is to be added before the name of BI virtual machines. | customPrefixNameBIDisk | A prefix is to be added before the name of BI Disks. | customPrefixNameQE | A prefix is to be added before the name of Query Engine virtual machines. | customPrefixNameQEDisk | A prefix is to be added before the name of Query Engine disks. | customPrefixNameKM | A prefix is to be added before the name of the Kyvos Manager virtual machine. | customPrefixNameKMDataDisk | A prefix is to be added before the name of the Kyvos Manager disk. | zone | The zone in which you want to deploy the resources. | bucketName | Enter the bucket name (if the bucket is not present, a new bucket would be created). | kyvosWorkDir | Provide the path of the Kyvos work directory. | kyvosClusterName | Name of the Kyvos cluster to be deployed. | bundleAccessKey | Key to access Kyvos bundle. | bundleSecretKey | The secret key for Kyvos bundle. | sshPublicKey | Text of the SSH public key for authentication. | sshPrivateKey | Enter the SSH private Key text of the pem file. NOTE: The text must be in base64 encoded. | kyvosLicenseFileValue | Enter a valid Kyvos license key. NOTE: It should be base64 encoded | additionalLabels | Labels to be added to the resources. |
Using existing Service AccountOnce resources are created, execute the following commands using the gcloud CLI to link the Kubernetes Service account to the IAM Service account. gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:PROJECT_ID.svc.id.goog[kyvos-monitoring/default]" gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:PROJECT_ID.svc.id.goog[kyvos-compute/default]" In the above-mentioned commands, replace the following: IAM_SA_NAME: The name of your new IAM service account. IAM_SA_PROJECT_ID: The project ID of your IAM service account. PROJECT_ID: The project ID of your Google Cloud.
Tip |
---|
Tip If you encounter any issue while processing the semantic model through Kubernetes, you cab debug the issue. For more details about debugging, see the Kubernetes documentation. |
|
...