Changelog

Third-party infrastructure dependency versions are listed by release on the Installation and Configuration page.

Cortex Charts version 6.3.3

October 21, 2022

NOTE

Cortex Fabric v6.3.1 and v6.3.2 were internal-only releases.

• Updated officially supported Kubernetes version to 1.23.
• TTL implemented for task logs/artifacts stored in Managed Content.

Cortex Charts version 6.3.0

August 8, 2022

• Security improvements have been made.
• Istio and Dex have been upgraded.

Cortex Charts version 6.2.2

June 21, 2022

• Security improvements have been made.

Cortex Charts version 6.2.1

May 21, 2022

• Defects and vulnerabilities have been addressed.
• Redbox Scuttle has been replaced with CognitiveScale’s C12E Scuttle.
• OEM key configuration for CData is supported.

Cortex Charts version 6.2.0

May 6, 2022

• Fabric Administrators are able to configure Fabric Managed Content to connect to Google Cloud Storage (GCS).
• Fabric Administrators can use Customer Managed Keys (CMK) (or public-keys) to encrypt Google Cloud Storage buckets.
• Fabric Administrators are able to configure role-based authorization (Workload Identity roles, service accounts) for enabling access to cloud storage buckets.  
• Fabric Administrators are able to configure TLS for data encryption in transit while accessing buckets in GCS.

Cortex-Charts version 6.1.1

March 30, 2022

NOTE

Version 6.1.0 has been omitted from this changelog as it was an initial version of the content in this official release.

• Support added for installation of Fabric on the Google Cloud Platform’s Google Kubernetes Engine.
• Periodic cleanup of activations and keys in Redis has been added to the Cortex Fabric infrastructure as a janitor job (cortex-synapse-janitor) when enabled in values.yaml.
• EKS platform has been upgraded to v1.21
• Support for Istio has been upgraded to version 1.12.x

Charts Docs

• Added: GKE Prerequisites page
• Updated: RUNBOOK: Installation and Configuration page
• Added documentation for using PodSecurity controllers within Kubernetes to restrict privilege escalation of pods within a namespace.

Cortex-Charts version 0.6.10.1

February 22, 2022

This is a documentation release with changes that include:

• New Kubernetes section with:

  • New content:
    • IP Address Range-CIDR
    • Encryption
  • Updated content:
    • Service Accounts

• Infrastructure Section Content:

  • New pages:
    • MongoDB
    • MinIO
    • Redis
  • Updates to:
    • Container Registry (previously Private registry)
  • Moved pages:
    • Persistence (moved to Kubernetes section)
    • Monitoring and Metrics (moved to Kubernetes section)
  • Platforms Section new pages:
    • AWS Section
      • AWS IRSA
    • GCP section
      • Workload Identity

NOTE

Some functionality described in this documentation release may not be available until the v0.6.11 code release.

Cortex-Charts version 0.6.9

October 13, 2021

• FEATURE_MAX_NAME_LENGTH environment variable added so users can change default for component names.
• Default REQUEST_LIMIT added for cortex-operator to truncate logs.
• Config var set for DEX to expire authRequest custom resources after 1 hour instead of the 25h default.
• Support for new version of the MongoDB subchart
• Migration job and instructions added to move data to new Mongodb cluster.
• Fabric now supports deploying Kubernetes resources (Agents/Skills/Tasks) into a targeted k8s namespace (i.e. cortex-compute).
• Updated packaged infrastructure subcharts for Dex, Docker-registry, Redis, external-dns, and Vault.

Known Issues

• DEX and API HPAs must be overridden/controlled separately from the cortex.autoscaleRules HPA.

Cortex-Charts version 0.6.8

August 19, 2021

• Cluster-autoscaler subchart has been updated to support newer k8s versions (compatible up to v1.20).
• Examples packaged with the Fabric6 Helm chart have been updated.
• Passwords on REDIS sentinels and nodes are now enabled by default.
• Use and configuration of HorizontalPodAutoscale rules are enabled for Cortex services
• A new synapse janitor CronJob has been added to Cortex Charts that runs on an interval and cleans up ALL activation records older than the time specified by the variable janitor.synapse.retentionDays.

Cortex-Charts version 0.6.7

June 30, 2021

Info

v0.6.6 was a hot-fix release.

• Registry sub-chart is now enabled by default.
• Support for istio 1.9.x has been added.
• Options have been added to set resource requests/limits on the k8s deployments/pods for cortex services.
• Dex storage now uses kubernetes CRD's instead of inmemory to allow for scaling.
• A new option has been added to helm chart values.yaml for enabling IRSA (cortex.irsa.enabled) .

Cortex-Charts version 0.6.5

May 11, 2021

Info

v0.6.4 was a hot-fix release.

• Improved Cortex Session REDIS implementation, using HSET and HMGET over GET/SET with multiple keys.
• Redis has been upgraded to v6.0.12.
• Minio has updated been updated (image tag: RELEASE.2021-03-26T00-00-41Z-35-g5cce9361b)
• Base resource limits on operator deployment have been increased to 2cpu/1Gi memory
• Security vulnerabilities across all Cortex services (source libs and OS patches) have been addressed.

Cortex-Charts version 0.6.3

April 7, 2021

Charts

• Activations can now be listed and filtered (using the API) by state, started before/after timestamp, and ended before/after timestamp.
• Packaged docker-registry subchart can be used with specified user/password; the Docker registry is exposed via Istio networking.

Operator (Bug Fixes)

• Active and queued counts in the default TaskPool are now correct and in sync.
• imagePullSecrets is now being applied as expected for daemons and jobs.
• Activations list is now scoped to Projects

Cortex-charts version 0.6.2

March 4, 2021

• Auth service now connects to Redis, so it can be scaled to enhance performance.
• Auth service was added to the health-check list in the Dashboard.
• Issue that caused intermittent 504 errors when using the UI and CLI was resolved.
• Agents with job failures are no longer stuck in a PENDING state.
• Support for Kubernetes Metric Server has been added to Fabric.

Cortex-charts version 0.6.1

December 16, 2020

• For improved performance, the cortex-api container has been updated to include the JAVA_TOOL_OPTIONS environment variable to provide more fine-grained control of that service. By default, the InitialHeapSize is set to 1GB, and the MaxHeapSize is set to 8GB.
• API service has been added to the Cortex system-health checks that are displayed in the Fabric Console.

Cortex-charts version 0.6.0

December 4, 2020

Cortex now supports enabling mTLS encryption for increased security in your cluster. By default Istio's authentication is set to permissive mode, but a sample configuration is now provided showing how to enable mTLS across the cluster by setting the mode to strict.

RabbitMQ has been removed from the dependencies stack.

Cortex-charts version 0.6.0-a.2

November 25, 2020

• Work has continued on optimizing authentication security.
• Styling and branding added to the SSO login page.

This release is a Preview. You may experience issues.

Cortex-charts version 0.6.0-a.1

November 4, 2020

Welcome to Cortex Fabric v6 Preview Release. We are in the process of making significant improvements to Cortex. Our Preview release includes:

• Vault has been made optional.
• Support for Vault injection to Cortex has been added.
• Dex has been added to the infrastructure to manage authentication for SSO.

Keep in mind that this is a Preview release, so you will likely encounter problems trying to do all things you are used to doing in Cortex Fabric. In particular Fabric Console, Fabric Docs, Cortex APIs, and Python Libraries are under construction. Please be patient, GREAT things are on the way!

Cortex-charts version 0.5.6

• Updated bl service dependencies due to CVE-2020-8244
• Metric events are now disabled internally
• Added a readiness probe to the agents service, so that it is now more fault tolerant
• Replication on the Postgres subchart has been disabled by default
• Helm version 3 is now fully supported
• Installation documentation has been improved and updated including adding a "Dependencies Compatibility Matrix"

Cortex-charts version 0.5.5

• Cortex-Charts Docs updates:
  • Logging
  • OpenShift 4.x installation
• Ingress resources are now fully compatible with Kubernetes 1.16

Breaking Changes

Cortex Fabric is releasing the Beta version of its new web-based Console that integrates Studio, Profiles, Admin Console, and Fabric Docs. Users can try the web versions of Profiles and Studio by logging in to Admin Console or they may continue using the desktop version of Studio: Agent Composer and Profile-of-One. This feature introduces the following breaking change to Cortex-Charts.

If your site-specific helm chart values.yaml file specifies a value for cortex.domains.api, it must ONLY contain protocol and subdomain:

cortex:
  domains:
    api: https://api

The installation documentation for helm chart versions 0.4.1 and prior instructed the user to specify a FQDN, which is no longer supported.

Note that cortex.domains.api defaults to https://api and specification in site-specific values.yaml is not required.

Cortex-charts version 0.5.4

• In this release, all Cortex services have been updated to address CVE vulnerabilities discovered during a routine security scan. If you are updating an existing cluster to this release, please see the note below about steps to take to avoid upgrade issues.
• This release also removes Elasticsearch and Kibana from our packaged components. These now must be installed and configured externally.

ALERT

Note before upgrading: In the 0.5.4 version of the Cortex Fabric helm chart, we have updated all of the subchart components included in the Cortex Fabric bundle which causes issues when trying to upgrade an existing cluster. Before upgrading to this version, run the following command to cleanup resource conflicts to allow the upgrade to succeed:

kubectl delete \
  -n cortex \
  deployments,services,serviceaccounts,jobs \
  cortex-docker-ingress-backend \
  cortex-nginx-ingress-backend \
  cortex-docker-ingress-default-backend \
  cortex-nginx-ingress-default-backend \
  cortex-docker-ingress-controller \
  cortex-nginx-ingress-controller \
  cortex-connection-type-loader

The expected output of the above command should be similar to the following (note that some errors in this output are acceptable):

deployment.extensions "cortex-nginx-ingress-default-backend" deleted
deployment.extensions "cortex-nginx-ingress-controller" deleted
service "cortex-nginx-ingress-default-backend" deleted
service "cortex-nginx-ingress-controller" deleted
serviceaccount "cortex-nginx-ingress-backend" deleted
Error from server (NotFound): deployments.extensions "cortex-docker-ingress-backend" not found
Error from server (NotFound): deployments.extensions "cortex-nginx-ingress-backend" not found
Error from server (NotFound): deployments.extensions "cortex-docker-ingress-default-backend" not found
Error from server (NotFound): deployments.extensions "cortex-docker-ingress-controller" not found
Error from server (NotFound): services "cortex-docker-ingress-backend" not found
Error from server (NotFound): services "cortex-nginx-ingress-backend" not found
Error from server (NotFound): services "cortex-docker-ingress-default-backend" not found
Error from server (NotFound): services "cortex-docker-ingress-controller" not found
Error from server (NotFound): serviceaccounts "cortex-docker-ingress-backend" not found
Error from server (NotFound): serviceaccounts "cortex-docker-ingress-default-backend" not found
Error from server (NotFound): serviceaccounts "cortex-nginx-ingress-default-backend" not found
Error from server (NotFound): serviceaccounts "cortex-docker-ingress-controller" not found
Error from server (NotFound): serviceaccounts "cortex-nginx-ingress-controller" not found

Cortex-charts version 0.5.3

ALERT

The default scale of RabbitMQ nodes (replicas) in the Cortex5 helm chart has been changed from 3 to 1.

If you have configured your Cortex5 deployment to use an externalized RabbitMQ instance and disabled the included RabbitMQ instance in the helm chart, take no action.

If you're concerned with the throughput of the packaged RabbitMQ cluster and wish to continue with the current scale, please edit your [override|cortex|values].yaml used for your helm deployment of the Cortex5 chart and set rabbitmq.replicas to 3 before running helm upgrade --install to upgrade your deployment of the Cortex5 helm chart.

If you're not concerned with your rabbitmq throughput and upgrade your Cortex5 deployment to the current helm release without explicitly setting the replicas of rabbitmq nodes, (after upgrading with helm) please shell into and run the following commands from within the remaining rabbitmq pod in your cluster via k9s or kubectl to force the rabbitmq cluster to no longer search for the missing rabbitmq pods:

rabbitmqctl forget_cluster_node rabbit@cortex-rmq-1.cortex-rmq-headless.cortex.svc.cluster.local
rabbitmqctl forget_cluster_node rabbit@cortex-rmq-2.cortex-rmq-headless.cortex.svc.cluster.local

For more information about scaling RabbitMQ in Helm go here.

Cortex-charts version 0.5.2

• Security vulnerabilities mitigated by updating the following infrastructure images, including:
  • Redis to version 5.0.7
  • pgbouncer to version 1.12
  • postgresql to version 11.4.0-debian-9-r0 -Configuration changes required to support Istio version 1.5

Cortex-charts version 0.5.1

• Added support for providing custom SMTP configurations.

Cortex-charts version 0.5.0

New features

• Support for podspec has been added to manage and customize action deployment resources such as: GPU, memory, or CPU. The podspec feature works by patching the Kubernetes PodTemplateSpec generated by Fabric.
• Support for non-root installation of the Cortex CLI
• Documentation has been added to cortex-charts for:
  • Autoscaling
  • Logging
  • Monitoring

Changed features

• For action deploy, options --memory --vcpu are no longer available. Instead use the [podspec] feature to customize pod resources for actions at runtime.
• When you use podspec for Kubernetes, action metrics are externalized from Cortex Console and will not be viewable there. To have access to logs and metrics you must use a third party aggregator like Grafana, Cloud Watch, or Azure Dashboards. Logging is delegated to a log aggregator defined on the cluster and will NOT be provided in Cortex Console.
• Profile-of-one has migrated from a single event loading paradigm to a bulk loader, which is able to process multiple profile events in a single job. The following is a summary of the changes that resulted from this shift in paradigm:
  • The underlying data model changed to easily allow the bulk insertions of profile attributes against the underlying Po1 database.
  • The way profiles are versioned changed, allowing multiple attributes to be updated per version.
  • The association between Profile Attribute and Specific Profile Schemas was removed, so the profiles no longer need to rebuild every time a Profile Schema changes.

Cortex-charts version 0.4.1

February 28, 2020

• For RedHat OpenShift 3.11 deployments, special privileges and/or run-as-root is not required to install and configure clusters under the OpenShift security context, non-root.
• Cortex alpine-based images have been deprecated in favor of more secure UBI8-based images for all deployments.

Cortex-charts version 0.4.0

February 18, 2020

• UBI images updated from ubi7 to ubi8 for improved security.
• Images updated to ensure that Redis sentinels are optimally configured.

ALERT

If you are upgrading from Helm chart version 0.3.1 or earlier to version 0.4.0 you may encounter the following issue:

UPGRADE FAILED
Error: persistentvolumeclaims “cortex-docker-registry” is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

To resolve this issue:

1. Find the storage class using kubectl get pvc cortex-docker-registry -n cortex -o yaml | grep storageClassName

   Example:

   kubectl get pvc cortex-docker-registry -n cortex -o yaml | grep storageClassName

   Output is: storageClassName: default

2. Edit the storageClassName by running this command: kubectl edit sc <storageClassName> appended with allowVolumeExpansion: true.

3. Confirm that the configuration change is applied by running: kubectl get sc <storageClassName> -o yaml | grep allowVolumeExpansion

   Example:

   kubectl get sc default -o yaml | grep allowVolumeExpansion

   Output is: allowVolumeExpansion: true

4. Restart the docker-registry pod in cortex.

5. Confirm the storage on the pvc is now increased to 100GB by running: kubectl get pvc cortex-docker-registry -n cortex

   Output is:

   NAME                     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   cortex-docker-registry   Bound    pvc-xyz   100Gi      RWO            default        160m

   For additional assistance with Kubernetes storage classes go to:

• Storage Classes
• Change the default StorageClass