Applies to: 
Kyvos Enterprise 
Kyvos Cloud (Managed Services on AWS) Kyvos Azure Marketplace
Kyvos AWS Marketplace Kyvos Single Node Installation (Kyvos SNI) Kyvos Free (Limited offering for AWS)
This section provides information on the validations that must be performed by the DevOps team after cluster deployment.
Ensure all EC2 instances are up and running
The cluster should be completed with BI Server, Query Engines, Web server, Kyvos Manager servers, and Bastion Host.
Login to the AWeb Server console then go to EC2 service and check all instances like BI Server, Query Engines, Web server, Kyvos Manager servers, and Bastion Host.
Ensure CFT deployed two BI Server, five Query Engines, one Web server, one Kyvos Manager server, and one Bastion Host
In the cluster, there should be two BI Server, five Query Engines, one Web server, one Kyvos Manager server, and one Bastion Host as Jump host.
Five Query Engines:
Two BI Servers:
One Web Server (secondary web portal, this instance will be used to run the Kyvos web portal):
One Kyvos Manager server (Primary web server, this instance will be used to run the Kyvos web portal and Kyvos Manager):
One Bastion Host:
Ensure BI servers are in different AZ’s
BI Servers should be in different availability zones as per SOC2 compliance.
Check both BI severs, it should be in different availability zone as in below screenshot:
Here, BI1 is in us-eat-2b AZ and BI2 is in us-east-2a AZ
Ensure all three QE servers are in AZ1 and the rest two QE in AZ2
Both QE Servers should be in different availability zones as per SOC2 requirements.
Three Query Engines should be in AZ1, and two Query Engines should be in AZ2.
Example, us-east-2a is AZ1 and us-ease-2b is AZ2.
Ensure Kyvos Manager & Web Server are in different AZ
Kyvos Manager server & Web Server should be in different availability zones as per SOC2 compliance. Kyvos Manager instance is the primary web portal & Web Server1 instance is the secondary web portal
Web Server1 instance should be in one region and Kyvos Manager instance should be in another region
Kyvos Manager instance:
Web Server1 instance:
On the above screenshots, the Kyvos Manager instance is in us-east-2b, and Web Server1 is in us-east-2a.
Autoscaling should be enabled on Bastion host
The Bastion host is a jump host from which Kyvos Manager portal is accessible by port forwarding. To access Kyvos Manager, the bastion host should be running. To keep access to Kyvos Manager, the bastion host is required to be up.
Refer to section Bastion Host autoscaling.
Ensure all EC2 instances are attached to a security group <To be removed, as per now it's not required>
The instances of the cluster should be in specific VPC and connected with its appropriate security group
Login to the AWS EC2 console and search cluster name, and apply the filter with the security group. Check all EC2 instances should be attached to a security group.
Check S3 replication is enabled If DR is enabled
If DR is enabled, then the S3 bucket of the Primary cluster should be available and replicated to the DR region.
In the primary cluster, S3 bucket and check under Management panel > Replication Rules and verify the S3 bucket replication region.
Ensure RDS read replica is set If DR in enabled (In case of DR enabled)
If DR is enabled, then the RDS of the Primary cluster will be replicated to the DR region.
Go to the Region where Disaster recovery (DR) is enabled, and check & verify the RDS replica status. RDS name should be as per the cluster name.
Select DB > Connectivity & Security:
Check Target group is healthy
If the Target group is healthy then it shows both KM (primary web server) and Web server are reachable and can respond.
Check the Target group of the cluster and check its health status.
For example, from the above screenshot, one webserver is up and running showing healthy and another webserver is unhealthy. But as per the validation step both “registered targets” should be healthy.
Check VPC and S3 is created in DR region (Applicable only when DR is opted)
If Disaster Recovery is applied to the cluster, then VPC and S3 data will be replicated to the DR region at the time of primary cluster creation.
Whenever the DR cluster is up, then the DR cluster will use replicated S3 bucket and VPC.
Go to the DR region, then select VPC. VPC will be created by cluster name.
Verify the S3 bucket, it should be created with the cluster name.
Check Secret Manager and RDS is replicated in DR (Applicable only when DR is opted)
After successfully creating the Cluster at the Primary region and if DR is applied, then the Secrets Manager key and RDS of the Primary region will also be replicated in the DR region.
RDS uses a secret manager key in the cluster. If DR is enabled, then the Secrets Manager key and RDS should be replicated in the DR region.
Secret Manager Screenshot:
RDS:
Verify user can login to Kyvos Manager portal
After successfully cluster creation login on Kyvos Manager with the help of SSH Port forwarding (Tunneling).
This step is to check & verify the accessibility of Kyvos Manager UI. The default username/password is admin/admin.
Refer the URL: Internal Kyvos Documentation
Verify desired number of BI server and QE are available on Kyvos Manager portal
On Kyvos Manager Portal, there is a cluster dashboard that contains the instance’s IP along with its Role (BI Server, Query Engines, Kyvos Manager, Web Server).
Check instances and number of BI and QE services along with its role & all services should be healthy (green)
Ensure license has been uploaded and verified from both Kyvos Manager and Kyvos UI
In the license, the allowed number of BI Servers and Query engines can be verified.
License verification from Kyvos Manager: Managing Kyvos License.
Verify user can login to Kyvos Portal
After successfully cluster creation login on Kyvos Portal
Verify completed jobs on Activity Monitor on Kyvos Portal (With Support team)
All activities of the Kyvos portal can be monitored from the Kyvos Portal.
Login on Kyvos Portal > Monitor > Completed processes.
Verify in KMS keys "DevOps AWS console user (Pramod, Humera, Vikas)" are added
In the current CloudFormation Template, the member who creates the cluster only can start the cluster. Other team members can only stop the cluster. To provide the access to team member then their AWS login username should be added to the KMS key.
Go to the AWS KMS (Key Management Service) console at Primary Region, then select customer-managed keys and select cluster KMS key and its key policy.
Go to the AWS KMS console at Disaster Recovery Region, then select customer-managed keys and select cluster KMS key and verify its key policy.
Verify Pem keys and EMR certificate are saved in central S3 bucket
The PEM key and EMR certificate are used in the cluster. The PEM keys are used to login on Bastion host and Kyvos Manager host. These keys are used by DevOps & Support team members. The DevOps team is responsible for saving the keys.
Following S3 bucket used to save the keys & EMR certificate
Example: S3: kyvos-devops/<region-name>/customer_data/<Stack Name>
Go to S3 bucket and search cluster name, verify PEM keys and EMR certificate in it.
No Error in Application logs after enabling TLS
When the cluster is deployed then application logs should be clear (without an error). To check application logs, login on BI. KM and Web server from terminal then verify the logs from following locations
Logs file Location:
Kyvos application Logs (KM Node & WS1 Node)
- Kyvos Manager logs on KM instance: /data/kyvos/installs/kyvos/jakarta/logs
- Kyvos user portal logs on KM instance: /data/kyvos/app/kyvos/jakarta/logs
- Kyvos Web Portal Logs on WS1 instance: /data/kyvos/app/kyvos/jakarta/webapps/kyvos/client/logs
BI server Logs (BI server Node)
/data/kyvos/app/kyvos/olapengine/logs
QE Logs (QE Node)
/data/kyvos/app/kyvos/queryengine/logs
Ensure WAF is enabled
WAF helps to protect the Applications or APIs against common web exploits and bots that may affect availability, compromise security, or consume excessive resources. Web Application Firewall should be enabled for Kyvos Environment.
Go to AWS WAF & Shield console, select WAF & Shield, then select WebACLs with cluster name.
Pre-validation
RDS Postgres version should be 13.6
Verify it from the CloudFormation template.
Post-validation
Kyvos Component version should be matched as per the release version
Verify it from the Kyvos Manager and Kyvos Web portal.
| Kyvos Manager Portal | Kyvos Web portal |
|---|---|
|
|
Load Balancer Configuration and rules should be proper
Added a Single Target Group from port 8443 for KM node and added rules
/kyvos/sql OR /kyvos/sql/* OR /kyvos/sql/sqlSSO OR /kyvos/sql/sqlSSO/*
Attaching screenshots of Load balancer configuration:
KM-ALB should be associated with a separate security group to provide an additional level of security.
For “Kyvos Manager ALB”, it should be associated with a separate security group and inbound rule as Protocol: TCP, Type: HTPPS, Port 443, Inbound: 103.250.170.125/32 (Impetus VLAN IP)
New KM-ALB SG: sg-088fa4b571423b5a3
So, we created a new security group and attached it to KM-ALB
The new security group should be allowed in the “KM instance” “security group” (sg-0bc034a5c0ed9e592) on port 9443 for Target Group health check success
Enable secret value under BI server property
Login to Kyvos Manager UI, and under the Properties search for USE_SECRET_STORE. It should be yes.
USE_SECRET_STORE = yes under global properties
Validate the license file for supported instance type series for Query Engine as per Managed Service requirement
Open the license file and search for
<QueryLayerComputeRates>r5:0.125</QueryLayerComputeRates>
Instance Type: r5.4xlarge
The value can be either M5 or R5
Enable DR parameter value
Depends during deployment: Enable or Disable
Webserver HA and Security Group Mapping
Added WS1 in Kyvos Manager with HA enable
After adding WS1, health checks are green
Health Checks
Verify the status of health checks after adding ALB security group to KM node and WS1 node’s security groups
Here both KM and WS1 has same/single security group attached
SMTP configuration for Kyvos Manager
SMTP details:
Server = smtp.office365.com
username = noreply-alerts.365@kyvos.io
Email address: noreply-alerts.365@kyvos.io
Password = testPassword
Port=587
TLS=TLS 1.2
In Kyvos Manager, under the Settings configure the SMTP using the given details.
After applying the changes, validate the email ID.
Tunneling should be made with port 9443 to KM then only, it will work
You will receive a verification link for validation. Click on it to validate
Verification link: https://127.0.0.1:9443/kyvosmanager/mailverify/noreply-alerts.365@kyvos.io
LDAP Integrations
Configure LDAP from Kyvos Manager UI under the Settings section and verify the same by logging in to Kyvos Manager and Kyvos Portal using Kyvos Support Team’s id and password
WAF rules for XMLA connection
Verify using MS Excel
Open new excel file Under “Data” click on “From Web” Now provide the URL and search.
Email Validation Page:
https://kyvosqa.free-trial.kyvosinsights.com/kyvos/
Addition rules found under Core Rule Set- SizeRestrictions_Cookie_HEADER, GenericLFI_BODY, GenericRFI_BODY
Cross Account Glue and required permission
Send a mail to Archit’s team to provide glue access to the Instance IAM role created like this-
On account ID 815559998352 in region us-east-2 give "arn:aws:iam::357135958305: role/InstanceIamRole-kyvosqa-815559998352" permission to access glue tables
After getting the access, the glue tables can be seen in kyvos portal
On-Screen Notification and SMTP use cases for Kyvos Portal
In Kyvos Manager, go to Java Options Configuration and append the below parameter in Additional Java Options.
-Dmail.smtp.starttls.enable=true -Dmail.debug=true -Dmail.smtp.ssl.protocols=TLSv1.2 
Under Security Configurations, update the LDAP values. Save it, apply, and then verify the same by logging in to Kyvos UI using support credentials.
Secret Manager Validation
Go to Secret Manager from the AWS console. Click on Retrieve secret value.
You will see the password is encrypted. To verify them, ping Mayuresh to decode the passwords. After conversion, validate them.
Sanity Suite
Run the sanity suite from Kyvos Manager using Kyvos credentials.
Validate the sanity results.
EMR Version
From the AWS console verify the EMR version, it should be 6.5 and Livy enabled.
EMR-Livy Configuration
Property for livy timeout livy.server.yarn.app-lookup-timeout ->1800s should be present on EMR under configuration section.
TLS –Certificate Validation
Verify using Sanity suite and Rest API <TBD>
TLS Port and Mapping
All the required ports for TLS (8443,8444,9443,9444) should be enabled in the server.xml file present in the Jakarta folder.
<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" redirectPort="8443" /> <Connector SSLEnabled="true" clientAuth="false" keystoreFile="/data/kyvos/app/kyvos/commons/config/ckstore/keystore.jks" keystorePass="bigdata" maxPostSize="10485760" maxThreads="500" port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" relaxedPathChars="[]" relaxedQueryChars="[]" scheme="https" secure="true" sslProtocol="TLSv1.2"> <Connector SSLEnabled="true" clientAuth="false" keystoreFile="" keystorePass="" maxThreads="200" port="9443" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true" sslProtocol="SSLv3" truststoreFile="" truststorePass="" usedfor="HTTPS_TLS"
Also, validate the same from the AWS console Target Group and ALB.
SQL Connectivity
Rule should be added in ALB: /kyvos/sqlOR/kyvos/sql/*OR/kyvos/sql/sqlSSOOR/kyvos/sql/sqlSSO/*
Single Instance Target Group is created with 8443 port for KM instance