Version: 0.5.6

Installation and Configuration

Info

The following instructions assume the user is proceeding from:

A completed platform setup.
A stable build of cortex-charts.

Dependencies Compatibility Matrix

Cortex Helm Chart	Helm	Kubernetes	Istio	RMQ	Redis	Mongo	Postgres	postgreSql	pgbouncer
0.5.5 and higher	2.x or 3.x	1.16 or higher	1.5.x	3.8.2	5.0.7	3.6.12	11.7.0	11.4.0-debian-9-r0	1.12
0.5.4 and lower	2.x	1.15 or lower	1.5.x	3.8.2	5.0.7	3.6.12	11.7.0	11.4.0-debian-9-r0	1.12

Prerequisites

Cortex Helm installation can be performed on the following platforms. Follow the links to view prerequisites that must be in place prior to performing the steps for Cortex Fabric helm installation.

AKS Installation Prerequisites
EKS Installation Prerequisites
Valid registered DCI license JWT and keys for docsToken and marketToken
SSL certificate and any intermediate certificate with associated private key files signed by a trusted CA
Utilities:
- azure-cli or aws-cli or gcloud CLI
- kubectl
- k9s
- helm
- jq
(Optional) Install Istio

Note: The instructions in this documentation currently use helm version 2 and might not fully work with the helm3 subcommands and options. Helm version 3 will be fully supported in a later release in accordance with the Helm Chart V2 deprecation timeline

Recommended Prerequisites

Read the following instructions for installing Helm and best practices guide.
- V2 Quickstart Guide
- The Chart Best Practices Guide
Read the following introduction to working with Kubernetes.
- Kubernetes documentation

Helm2 to Helm3 Migration

Existing clusters that have been deployed with Helm2 can be migrated using the guide and plugin provided by Helm.

There are known chart upgrade issues with helm after migrating an existing cluster to helm3.

It is recommended to schedule a maintenance window to resolve these issues (if present) in order to minimize downtime.

For any of the components (like redis or rabbitmq) that raise errors during an upgrade, deleting the resource using kubectl or k9s and rerunning the upgrade command will resolve the issue.

Cortex URLs

After your base-domain is set up, you will use it to access various Cortex services:

API - https://api.<base-domain> - Use this URL to connect Studio to a Dedicated Instance and for REST API calls and to authenticate to the Cortex CLI.
Console - https://console.<base-domain> - Use this URL to access the Console where you perform account and platform administration for your Cortex cluster. Log in to Console to invite other user's to Cortex, so they can set up their account login and credentials.

For instructions to install and login to Cortex Fabric tools, see https://cognitivescale.github.io/cortex-fabric/

You will require the following:

Cortex Invitation Code. The Cortex invitationCode is a unique alphanumeric token that must be provided when registering new accounts. This must be set in your cortex.yaml file (the overrides file) in step 1 of the installation instructions below. This code must be provided to users who are registering for new Cortex-Fabric accounts(following installation).

Install Cortex

Create a values file for deployment specific variables
Please follow the instructions here for using the example value.yaml below:

The Chart Best Practices Guide
Debugging Helm Charts
The example below provides you with a number of available settings. Not all settings are required. Use the ones that are applicable to your deployment environment. More detailed comments are available when you extract the chart release and view the packaged yaml values for the helm release you are deploying.

cortex.yaml - (click to expand)

It is recommended to explicitly set any cortex.*.password values for any production deployments. Any secret/password values not provided will be autogenerated during the initial installation and may rotate if installing via manifest (helm template | kubectl apply) rather than using helm's lifecycle management.

cortex:
    # invitationCode is a unique token that must be provided when regstering new accounts.
    invitationCode: invite_code
    # default account/user/password that is created during installation
    csadminAcct: cogscale
    csadminUsr: csadmin
    csadminPwd: csadminPassword
    domains:
      base: '<cluster_name>.<base_domain>'      
    smtp:
      email: 'cortex@cognitivescale.com'
      password: 'PASSWD'
      port: 465
      from: 'Cortex Activation <cortex@cognitivescale.com>'
      host: 'smtp.gmail.com'
    ssl:
      enabled: true        

  # REQUIRED ONLY IF MANAGED CONTENT IS BACKED BY S3
  # Also verify that "minio" component is set to false
    managedContent:
      S3_BUCKET: BucketName
      S3_ENDPOINT: https://s3.amazonaws.com
      S3_REGION: us-east-1
      accesskey: '<ACCESSKEY OF IAM USER WITH ACCESS TO S3>'
      secretkey: '<SECRETKEY OF IAM USER WITH ACCESS TO S3>'

  # REQUIRED ONLY IF THE DATABASES ARE EXTERNALIZED
    amqp:
      amqpUri: "rammitmqURI"
    mongodb:
      username: 'USERNAME'
      password: 'PASSWORD'
      metricsUri: "mongodb+srv://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@mongouri.mongodb.net/cortex-metrics?retryWrites=true&w=majority"
      graphUri: "mongodb+srv://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@mongouri.mongodb.net/cortex-graph?retryWrites=true&w=majority"
      mongoUri: "mongodb+srv://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@mongouri.mongodb.net/cortex-admin?retryWrites=true&w=majority"
    postgresql:
      password: 'POSTGRESPASSWORD'
    redis:
      env:
        REDIS_URI: 'redis://admin:$(REDIS_PASSWORD)@aws-us-east-1-portal.26.dblayer.com:34752'
      password: 'REDISPASSWORD'

  docs:
    cortexLicenseServer: "https://cortex_license_server"
  docker-ingress:
    enabled: true
  docker-registry:
    configData:
      auth:
        token:
         realm: 'https://api.&lt;cluster_name&gt;.&lt;base_domain>/v2/admin/docker/authenticate'    
  # CHANGE PERSISTENT VOLUME SIZE IF NOT USING S3 FROM THE DEFAULT SETTINGS OF 10Gi
    persistence:
      size: 200Gi           
  # REQUIRED IF DOCKER REGISTRY NEEDS TO BE BACKED BY S3
    secrets:
      s3:
        accessKey: '<ACCESSKEY OF IAM USER WITH ACCESS TO S3>'
        secretKey: '<SECRETKEY OF IAM USER WITH ACCESS TO S3>'
    storage: s3
    s3:
      region: us-east-1
      regionEndpoint: s3.us-east-1.amazonaws.com
      bucket: s3-bucket-docker-registry
      encrypt: true
      secure: true
  # END OF THE SETTINGS WHEN REGISTRY IS BACKED BY S3

  # REQUIRED WHEN POSTGRES IS EXTERNALIZED
  kong:
    pgbouncer:
      DatabasesHost: <postgres database host>
      DatabasesUser: postgres     

  # for OpenShift add the following to configure Route objects instead of Ingress
  ingress:
    type: Route        

  # Needed if AWS cluster autoscaling is requested
  cluster-autoscaler:
    enabled: true
    deployment:
      apiVersion: "apps/v1"
    cloudProvider: aws
    autoDiscovery:
      clusterName: clustername
    awsRegion: us-east-1
    awsAccessKeyID: "<AWS ACCESSKEY FOR AUTOSCALER IAM USER>"
    awsSecretAccessKey: "<AWS SECRET KEY FOR AUTOSCALER IAM USER> "

  # REQUIRED ONLY IF THE MONGODB IS EXTERNALIZED
  mongodb-replicaset:
    enabled: false

  # REQUIRED ONLY IF MANAGED CONTENT IS BACKED BY S3
  minio:
    enabled: false

  # REQUIRED ONLY IF THE PGSQL IS EXTERNALIZED
  postgresql:
    enabled: false

  # REQUIRED ONLY IF THE REDIS IS EXTERNALIZED
  redis:
    enabled: false

  # REQUIRED ONLY IF THE RABBITMQ IS EXTERNALIZED
  rabbitmq:
    enabled: false

  # REQUIRED ONLY IF RABBITMQ IS EXTERNALIZED for TLS support
  agents:
    env:
      NODE_TLS_REJECT_UNAUTHORIZED: '0'
  actions:
    env:
      NODE_TLS_REJECT_UNAUTHORIZED: '0'
  processorgateway:
    env:
      NODE_TLS_REJECT_UNAUTHORIZED: '0'
  connections:
    env:
      NODE_TLS_REJECT_UNAUTHORIZED: '0'
  metrics:
    env:
      NODE_TLS_REJECT_UNAUTHORIZED: '0'
  graph:
    env:
      NODE_TLS_REJECT_UNAUTHORIZED: '0'
  # REQUIRED IF Graph service is using an elastic cache redis
      REDIS_TYPE: "node"
      REDIS_CLUSTER_NODES: '["<Cluster config endpoint>"]'

  # CUSTOM IMAGE TAG DEFINED
  sessions:
    tag: 5.0.396-gf17b0f6

Setup Tiller and Helm. (required for helm2)

a. Apply RBAC to allow Tiller service to deploy helm charts.

kubectl apply -f https://raw.githubusercontent.com/CognitiveScale/dci-scripts/master/rbac.yaml

b. Initialize Helm/Tiller in Kubernetes

helm init \
--service-account tiller \
--upgrade \
--wait

Add Cortex Helm Repository to helm cache.

helm repo add cortex https://cognitivescale.github.io/cortex-charts/stable

Create a namespace for Cortex installation.
For cortex service: kubectl create namespace cortex
For cortex-compute service: kubectl create namespace cortex-compute

Create secrets for Cortex deploy. (Only needed when pulling Cortex images from Dockerhub)
For cortex service:
```
kubectl create secret docker-registry docker-login \
  --docker-server=<docker_registry> \
  --docker-username=<docker_username> \
  --docker-password=<docker_password> \
  -n cortex
```
NOTE: Use https://index.docker.io/v1/ for docker_registry, if using ECR or ACR provide entries in the value file for proper docker image locations.

Deploy Cortex via Helm chart.

helm upgrade --install cortex \
  cortex/cortex5 \
  --namespace cortex \
  -f cortex.yaml \
  --set cortex.ssl.cert="$(cat cert.crt)" \
  --set cortex.ssl.key="$(cat key.pem)"

To deploy a specific version of the Cortex Helm chart use the -version argument to specify the version number.

helm upgrade --install cortex \
  cortex/cortex5 \
  --namespace cortex \
  -f cortex.yaml \
  --set cortex.ssl.cert="$(cat cert.crt)" \
  --set cortex.ssl.key="$(cat key.pem)" \
  --version 0.1.55

Verify deployment status

Verify that the pods are deployed and stable in the namespace cortex was deployed to by using one of the two utilities (or any other kubernetes utility):

Your pods must be running and stable before moving on to the post deployment steps.

Post Deployment Steps

(Optional) Deploy the metrics-service to track node and pod metrics.

NOTE: Not needed for AKS as "metrics-server" is deployed as part of the default K8s services

helm install stable/metrics-server \
  --name metrics-server \
  --namespace metrics \
  --set args={"--kubelet-insecure-tls=true,--kubelet-preferred-address-types=InternalIP\,Hostname\,ExternalIP"}

(Only required if external-dns is not configured via cortex.yaml) Create A-record or CNAME DNS entry for * (/ wildcard) that points to the deployed ingress controller public IP address AND create A-record or CNAME DNS entry for "private-registry" that points to the secondary docker-ingress controller public IP address.
Identify docker-ingress service controller public IP address and use it to configure the Route 53 or DNS mapping records.
Run:
```
$ kubectl get svc -n cortex | grep cortex-nginx-ingress
```
Example Response:
```
cortex-nginx-ingress-controller        LoadBalancer   10.230.1.198   23.99.226.124   80:31839/TCP,443:30453/TCP              7h56m
cortex-nginx-ingress-default-backend   ClusterIP      10.230.1.157   <none>          80/TCP                                  7h56m
```
- For AKS
- For EKS
- For GKS (GCR) - TBD
- for OpenShift - TBD

(FOR AWS) Create the cortex-aws-s3 secret needed by managed content:

Encode the AWS Access key and Secret key with base64
```
$ echo -n "AKIAIOSFODNN7EXAMPLE" | base64
```
Output is similar to this: QUtJQUlPU0ZPRE5ON0VYQU1QTEU=
```
$ echo -n "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" | base64
```
Output is similar to this: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQ==
```
$ echo -n "changeme" | base64
```
Output is similar to this: Y2hhbmdlbWU=

Put the values in the cortex-aws-s3.yaml file:

apiVersion: v1
data:
  content-key: QUtJQUlPU0ZPRE5ON0VYQU1QTEU=
  content-secret: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQ==
  docs-key: Y2hhbmdlbWU=
  docs-secret: Y2hhbmdlbWU=
  minio-key: Y2hhbmdlbWU=
  minio-secret: Y2hhbmdlbWU=
kind: Secret
metadata:
  name: cortex-aws-s3
  namespace: cortex
type: Opaque

Apply the secret by
- deleting it: $ kubectl delete secret cortex-aws-s3 -n cortex
- then recreating it: $ kubectl apply -f cortex-aws-s3.yaml

Enable authentication required for the cortex-compute namespace to pull from private-registry:
NOTE
The pre-requisite for this script is that jq needs to be installed on the system where it is being run.

Execute this command to update "docker-login" secret for cortex and cortex-compute namespaces.

In *nix terminal:

wget -O create-token-actions.sh https://raw.githubusercontent.com/CognitiveScale/dci-scripts/master/create-token-actions.sh && bash ./create-token-actions.sh kubectl docker-login

In Mac terminal:

wget -O create-token-actions.sh "https://raw.githubusercontent.com/CognitiveScale/dci-scripts/master/create-token-actions.sh" && bash ./create-token-actions.sh kubectl docker-login

Upgrade Cortex Fabric Helm chart

For deployments use helm repo add cortex https://cognitivescale.github.io/cortex-charts/stable

helm upgrade \
 --install \
 --namespace cortex cortex cortex/cortex5 \
 -f cortex.yaml

Next steps

Install logging and monitoring tools.

Info

#Dependencies Compatibility Matrix

#Prerequisites

#Recommended Prerequisites

#Helm2 to Helm3 Migration

#Cortex URLs

#Install Cortex

#Verify deployment status

#Post Deployment Steps

NOTE

#Upgrade Cortex Fabric Helm chart

#Next steps