OpenShift

The following details instructions for configuring and installing Cortex Fabric on top of RedHat OpenShift 3.11 or 4.x.

Prerequisites to install OpenShift 3.11 or 4.x

• An OpenShift/RedHat Account
• 2 DNS Hosted zones
  • one private for internal name resolution
  • one public for external access
    • Make sure your VPC has “DNS resolution” set to true.
• Valid certificates for your public DNS (from Letsencrypt or Namecheap)
• A dedicated ssh key-pair for the cluster
• Security group defined for OpenShift
• A RHN Subscription with “View/Edit all“ privilege

OpenShift 3.11/4.x Installation-Configuration Guide

The following instructions assume:

• The user is proceeding from the stable build
• All infrastructure prerequisites have been completed
• Helm is not being used (Helm template is created below)

Note: Deploying via helm template is non-standard and circumvents the Helm Hooks the Cortex Fabric chart uses to autogenerate and manage any secret values not set when initially deploying Cortex Fabric and all of the packaged infrastructure/subcharts, such as mongodb, postgresql, minio, redis, and rabbitmq.

Pre-installation steps

1. Create OpenShift projects.

   oc new-project cortex

   oc new-project cortex-compute

2. Disable ephemeral storage checks and evictions by updating the value in the file, replace feature-gates with information below. (Not needed on OpenShift 4.x)

• Open the config file:

  oc edit cm node-config-master -n openshift-node
  -

• Find and edit this line in the .yaml file that opens by adding ,LocalStorageCapacityIsolation=false:

  RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true,LocalStorageCapacityIsolation=false

Helm Template and Cortex Installation

Steps 2-5 may be performed by a non-privileged user; troubleshooting and step 6 must be performed by a user with system administrator privileges.

1. Provide sys admin with privileged RBAC files and apply them:

   helm template cortex5 -x templates/security/cluster-role.yaml > template-priv.yaml && \
   helm template cortex5 -x templates/security/cluster-binding-cortex.yaml >> template-priv.yaml && \
   helm template cortex5 -x templates/security/cluster-binding-compute.yaml >> template-priv.yaml

   oc apply -f template-priv.yaml -n cortex

2. Create the Helm template.

   helm template cortex5 -f cortex5/examples/openshift/values-cortex-openshift.yaml -f local_override.yaml --namespace cortex -n cortex > template.yaml

3. Apply the template to your project.

   oc apply -f template.yaml

4. Create the docker-login secret.

   oc create secret generic docker-login --from-file=.dockerconfigjson=dockerconfig.json --type=kubernetes.io/dockerconfigjson -n cortex

   TROUBLESHOOTING: This requires sys admin privileges After spinning up a vanilla OpenShift cluster, while you are installing Cortex, you may encounter the following issues. The corrections must be complete by a System Administrator:

   • Error Msg #1:

     create Pod cortex-postgres-master-0 in StatefulSet cortex-postgres-master failed error: pods "cortex-postgres-master-0" is forbidden: unable to validate against any security context constraint: [fs│
            │roup: Invalid value: []int64{1001}: 1001 is not an allowed group spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1000400000, 1000409999]]

     Correct the above error by running the following:

     oc adm policy add-scc-to-group nonroot system:serviceaccounts:cortex

   • Error Msg #2:

     Warning  FailedScheduling  4m48s (x14 over 5m2s)  default-scheduler  0/5 nodes are available: 1 node(s) had no available volume zone, 4 node(s) didn't match node selector.

     Correct the above error by running the following:

     oc patch namespace cortex -p '{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'

5. Delete the Mongo test pod that is created during installation.

   kubectl delete pod cortex-mongodb-replicaset-test -n cortex

6. Create the Private Registry token.

   curl https://cognitivescale.github.io/cortex-charts/utilities/create-token-actions.sh | bash

7. If the external-dns subchart component has not been configured during the helm install, you must configure the DNS for your ingress resources.

Upgrades

When using helm template for installing/upgrading Cortex Fabric (and other Helm Charts), it is recommended to set all password/secret credentials for external services (such as mongodb and postgresql) listed under the cortex config block of overrides for the deployment. This keeps these secrets/credentials from being rotated each time a manifest is built via helm template and potentially locking Cortex services access to infrastructure components between releases. Before upgrading existing deployments, please extract these values from the existing secret resources deployed via kubectl/k9s (or your preferred kubernetes management utilities.) An example for retrieving the cortex.postgresql.password value in both a running postgresql pod's environment and the base64 encoded value from the secret resource:

kubectl exec -it cortex-postgres-master-0 -n cortex -- bash -c  "env | grep -i pass"
kubectl get secret cortex-postgresql -n cortex -o yaml | yq r - data["postgresql-password"]