Document toolboxDocument toolbox

DevOps Configurations, Validations, and Actions

Owned by Nidhi Bhandari (Unlicensed)
Last updated: Nov 09, 2023 by NIDHI BHANDARI
10 min read

Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

This section provides information on the validations that must be performed by the DevOps team after cluster deployment.

Ensure all EC2 instances are up and running.

The cluster should be completed with BI Server, Query Engines, Web server, Kyvos Manager servers, and Bastion Host.

Login to the AWeb Server console then go to EC2 service and check all instances like BI Server, Query Engines, Web server, Kyvos Manager servers, and Bastion Host.

Ensure CFT deployed two BI Server, five Query Engines, one Web server, one Kyvos Manager server, and one Bastion Host 

In the cluster, there should be two BI Server, five Query Engines, one Web server, one Kyvos Manager server, and one Bastion Host as Jump host.

Five Query Engines:

Two BI Servers:

One Web Server (secondary web portal, this instance will be used to run the Kyvos web portal):

One Kyvos Manager server (Primary web server, this instance will be used to run the Kyvos web portal and Kyvos Manager):

One Bastion Host:

Ensure BI servers are in different AZ’s

BI Servers should be in different availability zones as per SOC2 compliance.

Check both BI severs, it should be in different availability zone as in below screenshot:

Here, BI1 is in us-eat-2b AZ and BI2 is in us-east-2a AZ

Ensure all three QE servers are in AZ1 and the rest two QE in AZ2

Both QE Servers should be in different availability zones as per SOC2 requirements.

Three Query Engines should be in AZ1, and two Query Engines should be in AZ2.

Example, us-east-2a is AZ1 and us-ease-2b is AZ2.

Ensure Kyvos Manager & Web Server are in different AZ

Kyvos Manager server & Web Server should be in different availability zones as per SOC2 compliance. Kyvos Manager instance is the primary web portal & Web Server1 instance is the secondary web portal

Web Server1 instance should be in one region and Kyvos Manager instance should be in another region

Kyvos Manager instance:

Web Server1 instance:

On the above screenshots, the Kyvos Manager instance is in us-east-2b, and Web Server1 is in us-east-2a.

Autoscaling should be enabled on Bastion host

The Bastion host is a jump host from which Kyvos Manager portal is accessible by port forwarding. To access Kyvos Manager, the bastion host should be running. To keep access to Kyvos Manager, the bastion host is required to be up.

Refer to section Bastion Host autoscaling.

Ensure all EC2 instances are attached to a security group <To be removed, as per now it's not required>

The instances of the cluster should be in specific VPC and connected with its appropriate security group

Login to the AWS EC2 console and search cluster name, and apply the filter with the security group. Check all EC2 instances should be attached to a security group.

Check S3 replication is enabled If DR is enabled

If DR is enabled, then the S3 bucket of the Primary cluster should be available and replicated to the DR region.

In the primary cluster, S3 bucket and check under Management panel > Replication Rules and verify the S3 bucket replication region.

Ensure RDS read replica is set If DR in enabled (In case of DR enabled)

If DR is enabled, then the RDS of the Primary cluster will be replicated to the DR region.

Go to the Region where Disaster recovery (DR) is enabled, and check & verify the RDS replica status. RDS name should be as per the cluster name.

Select DB > Connectivity & Security:

Check Target group is healthy

If the Target group is healthy then it shows both KM (primary web server) and Web server are reachable and can respond.

Check the Target group of the cluster and check its health status.

For example, from the above screenshot, one webserver is up and running showing healthy and another webserver is unhealthy. But as per the validation step both “registered targets” should be healthy.

Check VPC and S3 is created in DR region (Applicable only when DR is opted)

If Disaster Recovery is applied to the cluster, then VPC and S3 data will be replicated to the DR region at the time of primary cluster creation.

Whenever the DR cluster is up, then the DR cluster will use replicated S3 bucket and VPC.

Go to the DR region, then select VPC. VPC will be created by cluster name.

Verify the S3 bucket, it should be created with the cluster name.

Check Secret Manager and RDS is replicated in DR (Applicable only when DR is opted)

After successfully creating the Cluster at the Primary region and if DR is applied, then the Secrets Manager key and RDS of the Primary region will also be replicated in the DR region.

RDS uses a secret manager key in the cluster. If DR is enabled, then the Secrets Manager key and RDS should be replicated in the DR region.

Secret Manager Screenshot:

RDS:

Verify user can login to Kyvos Manager portal

After successfully cluster creation login on Kyvos Manager with the help of SSH Port forwarding (Tunneling).

This step is to check & verify the accessibility of Kyvos Manager UI. The default username/password is admin/admin.

Refer the URL: Internal Kyvos Documentation

Verify desired number of BI server and QE are available on Kyvos Manager portal

  1. On Kyvos Manager Portal, there is a cluster dashboard that contains the instance’s IP along with its Role (BI Server, Query Engines, Kyvos Manager, Web Server).

  2. Check instances and number of BI and QE services along with its role & all services should be healthy (green)

  3. Ensure license has been uploaded and verified from both Kyvos Manager and Kyvos UI

  4. In the license, the allowed number of BI Servers and Query engines can be verified.

  5. License verification from Kyvos Manager: Managing Kyvos License.

Verify user can login to Kyvos Portal

  1. After successfully cluster creation login on Kyvos Portal

  2. Verify completed jobs on Activity Monitor on Kyvos Portal (With Support team)

  3. All activities of the Kyvos portal can be monitored from the Kyvos Portal.

  4. Login on Kyvos Portal > Monitor > Completed processes.

  5. Verify in KMS keys "DevOps AWS console user (Pramod, Humera, Vikas)" are added

  6. In the current CloudFormation Template, the member who creates the cluster only can start the cluster. Other team members can only stop the cluster. To provide the access to team member then their AWS login username should be added to the KMS key.

  7. Go to the AWS KMS (Key Management Service) console at Primary Region, then select customer-managed keys and select cluster KMS key and its key policy.

  8. Go to the AWS KMS console at Disaster Recovery Region, then select customer-managed keys and select cluster KMS key and verify its key policy.

  9. Verify Pem keys and EMR certificate are saved in central S3 bucket

  10. The PEM key and EMR certificate are used in the cluster. The PEM keys are used to login on Bastion host and Kyvos Manager host. These keys are used by DevOps & Support team members. The DevOps team is responsible for saving the keys.

  11. Following S3 bucket used to save the keys & EMR certificate
    Example: S3: kyvos-devops/<region-name>/customer_data/<Stack Name>

  12. Go to S3 bucket and search cluster name, verify PEM keys and EMR certificate in it.

  13. No Error in Application logs after enabling TLS

  14. When the cluster is deployed then application logs should be clear (without an error). To check application logs, login on BI. KM and Web server from terminal then verify the logs from following locations.
    Logs file Location:

Kyvos application Logs (KM Node & WS1 Node) 

  1. Kyvos Manager logs on KM instance: /data/kyvos/installs/kyvos/jakarta/logs

  2. Kyvos user portal logs on KM instance: /data/kyvos/app/kyvos/jakarta/logs

  3. Kyvos Web Portal Logs on WS1 instance: /data/kyvos/app/kyvos/jakarta/webapps/kyvos/client/logs

 BI server Logs (BI server Node) 

/data/kyvos/app/kyvos/olapengine/logs

 QE Logs (QE Node) 

/data/kyvos/app/kyvos/queryengine/logs

Ensure WAF is enabled

WAF helps to protect the Applications or APIs against common web exploits and bots that may affect availability, compromise security, or consume excessive resources. Web Application Firewall should be enabled for Kyvos Environment.

Go to AWS WAF & Shield console, select WAF & Shield, then select WebACLs with cluster name.

Pre-validation

RDS Postgres version should be 13.6

Verify it from the CloudFormation template.

Post-validation

Kyvos Component version should be matched as per the release version

Verify it from the Kyvos Manager and Kyvos Web portal.

Kyvos Manager Portal

Kyvos Web portal

Kyvos Manager Portal

Kyvos Web portal

Load Balancer Configuration and rules should be proper

Added a Single Target Group from port 8443 for KM node and added rules

/kyvos/sql OR /kyvos/sql/* OR /kyvos/sql/sqlSSO OR /kyvos/sql/sqlSSO/*

Attaching screenshots of Load balancer configuration:

KM-ALB should be associated with a separate security group to provide an additional level of security.

For “Kyvos Manager ALB”, it should be associated with a separate security group and inbound rule as Protocol: TCP, Type: HTPPS, Port 443, Inbound: 103.250.170.125/32 (Impetus VLAN IP)
New KM-ALB SG:   sg-088fa4b571423b5a3

So, we created a new security group and attached it to KM-ALB

The new security group should be allowed in the “KM instance” “security group” (sg-0bc034a5c0ed9e592) on port 9443 for Target Group health check success

Enable secret value under BI server property

Login to Kyvos Manager UI, and under the Properties search for USE_SECRET_STORE. It should be yes.

USE_SECRET_STORE = yes under global properties

Validate the license file for supported instance type series for Query Engine as per Managed Service requirement

Open the license file and search for

<QueryLayerComputeRates>r5:0.125</QueryLayerComputeRates>

Instance Type: r5.4xlarge

The value can be either M5 or R5

Enable DR parameter value

Depends during deployment: Enable or Disable

Webserver HA and Security Group Mapping

Added WS1 in Kyvos Manager with HA enable

After adding WS1, health checks are green

Health Checks

Verify the status of health checks after adding ALB security group to KM node and WS1 node’s security groups
Here both KM and WS1 has same/single security group attached

SMTP configuration for Kyvos Manager

SMTP details:

Server = smtp.office365.com
username = noreply-alerts.365@kyvos.io
Email address: noreply-alerts.365@kyvos.io
Password = testPassword
Port=587 
TLS=TLS 1.2

In Kyvos Manager, under the Settings configure the SMTP using the given details.

After applying the changes, validate the email ID.

Tunneling should be made with port 9443 to KM then only, it will work

You will receive a verification link for validation. Click on it to validate

Verification link: https://127.0.0.1:9443/kyvosmanager/mailverify/noreply-alerts.365@kyvos.io

LDAP Integrations

Configure LDAP from Kyvos Manager UI under the Settings section and verify the same by logging in to Kyvos Manager and Kyvos Portal using Kyvos Support Team’s id and password

WAF rules for XMLA connection

Verify using MS Excel

Open new excel file Under “Data” click on “From Web” Now provide the URL and search.

Email Validation Page:

https://kyvosqa.free-trial.kyvosinsights.com/kyvos/

Addition rules found under Core Rule Set- SizeRestrictions_Cookie_HEADER, GenericLFI_BODY, GenericRFI_BODY

Cross Account Glue and required permission

Send a mail to Archit’s team to provide glue access to the Instance IAM role created like this-

On account ID 815559998352 in region us-east-2 give "arn:aws:iam::357135958305: role/InstanceIamRole-kyvosqa-815559998352" permission to access glue tables

After getting the access, the glue tables can be seen in kyvos portal

On-Screen Notification and SMTP use cases for Kyvos Portal

In Kyvos Manager, go to Java Options Configuration and append the below parameter in Additional Java Options.
-Dmail.smtp.starttls.enable=true -Dmail.debug=true -Dmail.smtp.ssl.protocols=TLSv1.2

Under Security Configurations, update the LDAP values. Save it, apply, and then verify the same by logging in to Kyvos UI using support credentials.

Secret Manager Validation

Go to Secret Manager from the AWS console. Click on Retrieve secret value.

You will see the password is encrypted. To verify them, ping Mayuresh to decode the passwords. After conversion, validate them.

Sanity Suite

Run the sanity suite from Kyvos Manager using Kyvos credentials.

Validate the sanity results.

EMR Version

From the AWS console verify the EMR version, it should be 6.5 and Livy enabled.

EMR-Livy Configuration

Property for livy timeout livy.server.yarn.app-lookup-timeout ->1800s should be present on EMR under configuration section.

TLS –Certificate Validation

Verify using Sanity suite and Rest API <TBD>

TLS Port and Mapping

All the required ports for TLS (8443,8444,9443,9444) should be enabled in the server.xml file present in the Jakarta folder.

<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"               redirectPort="8443" /> <Connector SSLEnabled="true" clientAuth="false" keystoreFile="/data/kyvos/app/kyvos/commons/config/ckstore/keystore.jks" keystorePass="bigdata" maxPostSize="10485760" maxThreads="500" port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" relaxedPathChars="[]" relaxedQueryChars="[]" scheme="https" secure="true" sslProtocol="TLSv1.2"> <Connector SSLEnabled="true" clientAuth="false" keystoreFile="" keystorePass="" maxThreads="200" port="9443" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true" sslProtocol="SSLv3" truststoreFile="" truststorePass="" usedfor="HTTPS_TLS"

 Also, validate the same from the AWS console Target Group and ALB.

SQL Connectivity

Rule should be added in ALB: /kyvos/sqlOR/kyvos/sql/*OR/kyvos/sql/sqlSSOOR/kyvos/sql/sqlSSO/*

Single Instance Target Group is created with 8443 port for KM instance

Copyright Kyvos, Inc. All rights reserved.