High Availability

This is a guide to setting up High Availability (HA) for Cortex Fabric.

Cortex Fabric recommended setup supports HA out-of-the-box. The recommendations below should serve as reminders for the optimal setup.

To fetch the example packaged with a specific release of the Fabric6 Helm Chart use the helm pull command to pull and extract the Fabric6 chart.

helm pull cortex/fabric6 --untar --destination ./

Recommended Configuration

• Scale all Cortex services to 3
• MongoDB scaled to 3 replicas (if not externally managed)
• MinIO Gateway scaled to 3 replicas
• Redis scaled with 3 Sentinels (if not externally managed)
• Cloud Storage configured for content, amp, profiles (e.g. S3)
• IRSA support enabled and configured
• The persistence for the packaged Docker-registry set to use cloud storage (e.g. S3)
• MinIO disabled (If not externally managed)
• Cluster-autoscaler is enabled

Example Configuration

The Fabric6 Helm Chart is packaged with an example yaml file (examples/values-cortex-ha.yml) for the following recommended HA configuration. The example below is for AWS S3 platform. Your configuration may differ based on the platform and storage solutions you are using.

values-cortex-ha.yml - (click to expand)

cortex:
  irsa:
    enabled: true
  managedContent:
    MANAGED_CONTENT_BUCKET: managed-content
    AMP_BUCKET: amp
    PROFILES_BUCKET: profiles
    S3_ENDPOINT: https://s3.amazonaws.com
    S3_REGION: us-east-1
    # access/secret keys need to be provided if IRSA is not enabled
  #    accesskey: ''
  #    secretkey: ''
  mongodb:
    mongoUri: "mongodb://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@cortex-mongodb-replicaset-client.cortex.svc.cluster.local:27017/cortex_services?authSource=admin&replicaSet=rs0"
  redis:
    redisUri: 'redis://:$(REDIS_PASSWORD)@cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local:6379'
    env:
      REDIS_TYPE: 'sentinel'
      REDIS_NAME: 'mymaster'
      REDIS_SENTINELS: '[{"host":"cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local","port":26379},{"host":"cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local","port":26379},{"host":"cortex-redis-node-1.cortex-redis-headless.cortex.svc.cluster.local","port":26379},{"host":"cortex-redis-node-2.cortex-redis-headless.cortex.svc.cluster.local","port":26379}]'
      REDIS_MODE: 'sentinel'
      REDIS_SENTINEL_MASTER: 'mymaster'
      REDIS_SENTINEL_NODES: 'cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local:26379,cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local:26379,cortex-redis-node-1.cortex-redis-headless.cortex.svc.cluster.local:26379,cortex-redis-node-2.cortex-redis-headless.cortex.svc.cluster.local:26379'
      REDIS_SENTINEL_PASSWORD: '$(REDIS_PASSWORD)'
auth:
  replicaCount: 3
accounts:
  replicas: 3
api:
  replicas: 3
catalog:
  replicas: 3
connections:
  replicas: 3
console:
  replicas: 3
models:
  replicas: 3
operator:
  replicas: 3
processorgateway:
  replicas: 3
dex:
  replicas: 3
mongodb-replicaset:
  replicas: 3
minio:
  enabled: false
redis:
  sentinel:
    enabled: true
  cluster:
    enabled: true
    slaveCount: 3
cluster-autoscaler:
  enabled: true
docker-registry:
  enabled: true
  secrets:
    s3:
      accessKey: ''
      secretKey: ''
  storage: s3
  s3:
    region: us-east-1
    regionEndpoint: s3.amazonaws.com
    bucket: docker-registry
    encrypt: true
    secure: true
  persistence:
    enabled: false