Upgrades and Compatibility

This is a guide to the steps required for upgrading the Fabric6 Helm chart.

Upgrade Cortex Fabric Helm chart

The general process of upgrading the Fabric6 Helm chart is described below.

Prerequisites

Before running the upgrade command in kubectl:

• Stop the operator by running:

  kubectl scale deployment cortex-operator --namespace cortex --replicas=0

• Delete any task-pools by running:

  kubectl delete taskpool default --namespace cortex

Upgrade

For deployments use helm repo add cortex https://cognitivescale.github.io/cortex-charts/stable

helm upgrade \
 --install \
 --namespace cortex cortex cortex/cortex5 \
 -f cortex.yaml

Upgrade Notes for Cortex Fabric Helm chart 0.6.9

MongoDB Upgrade and Data Migration

With the 0.6.9 release of the Fabric6 Helm chart, a newer version of the mongodb helm chart is packaged along with a job to migrate the data from the existing mongodb-replicaset instance to the newer mongodb instance.

New clusters should be able to immediately switch over to use the newer Mongodb chart.

1. Run the following commands to add the required annotations and label to the cortex-mongodb secret

• kubectl annotate -n cortex secret cortex-mongodb meta.helm.sh/release-name=cortex

• kubectl annotate -n cortex secret cortex-mongodb meta.helm.sh/release-namespace=cortex

• kubectl label -n cortex secret cortex-mongodb app.kubernetes.io/managed-by=Helm

1. Upgrade the Helm deployment to initialize the new Mongodb and begin the data migration job.
2. Wait for the cortex-mongo-janitor job to complete.
3. Update the Mongo uri used by Cortex services to connect to the new Mongodb service (NOTE: Mongo uri values like $() must be changed to ${} when using Vault injection):

   cortex:
     mongodb:
       mongoUri: "mongodb://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@cortex-mongodb-headless.cortex.svc.cluster.local:27017/cortex_services?authSource=cortex_services&replicaSet=rs0"

4. Upgrade the Helm deployment to switch the Cortex services over to use the new Mongodb instance.
5. Verify that all data is present (you may want to delete the cortex-mongo-janitor job and upgrade to force it to rerun and sync any changes that happened between the last job run and switching to the new uri).
6. (Optional) Disable the old mongodb-replicaset deployment and data migration job in the Fabric6 chart overrides. (This subchart will be removed in a future release.)

   mongodb-replicaset:
     enabled: false
   janitor:
     mongo:
       enabled: false

Compute Namespace

The Fabric6 Helm chart supports deploying/managing user created resources into a targeted Kubernetes namespace (e.g. cortex-compute).

• Set the following Helm chart override:

  cortex:
    # cortex.computeNamespace: the k8s namespace that Fabric custom resources are created in
    computeNamespace: "cortex-compute"

• To create the cortex-compute namespace and add any required labels or imagePullSecrets (e.g. enabling Istio injection and copying over the secret to access the private-registry/annotating the default serviceAccount) run the following commands:

  • kubectl create ns cortex-compute

  • kubectl label namespace cortex-compute istio-injection=enabled

  • kubectl get secret docker-login-pr --namespace=cortex --export -o yaml |  kubectl apply --namespace=cortex-compute -f -

• Upgrade the Fabric6 Helm chart to enforce the cortex.computeNamespace setting.

• To move any already deployed Agents and Skills into the new cortex-compute namespace run the following commands:

  • Get a list of currently deployed resources:

    kubectl get agents,skills -o json --namespace cortex  > /tmp/cortexResources-old.json

  • Make a copy of the resource list with the namespace changed:

    cat /tmp/cortexResources-old.json | jq '.items[].metadata.namespace = "cortex-compute"' > /tmp/cortexResources-new.json

  • Delete the resources in the old namespace:

    kubectl delete --wait -f /tmp/cortexResources-old.json

  • Recreate the resources in the new namespace:

    kubectl create -f /tmp/cortexResources-new.json