High Availability
This is a guide to setting up High Availability (HA) for Cortex Fabric.
Cortex Fabric recommended setup supports HA out-of-the-box. The recommendations below should serve as reminders for the optimal setup.
To fetch the example packaged with a specific release of the Fabric6 Helm Chart use the helm pull command to pull and extract the Fabric6 chart.
helm pull cortex/fabric6 --untar --destination ./
Recommended Configuration
- Scale all Cortex services to 3
- MongoDB scaled to 3 replicas (if not externally managed)
- MinIO Gateway scaled to 3 replicas
- Redis scaled with 3 Sentinels (if not externally managed)
- Cloud Storage configured for content, amp, profiles (e.g. S3)
- IRSA support enabled and configured
- The persistence for the packaged Docker-registry set to use cloud storage (e.g. S3)
- MinIO disabled (If not externally managed)
- Cluster-autoscaler is enabled
Example Configuration
The Fabric6 Helm Chart is packaged with an example yaml file (examples/values-cortex-ha.yml
) for the following recommended HA configuration. The example below is for AWS S3 platform. Your configuration may differ based on the platform and storage solutions you are using.
values-cortex-ha.yml - (click to expand)
cortex: irsa: enabled: true managedContent: MANAGED_CONTENT_BUCKET: managed-content AMP_BUCKET: amp PROFILES_BUCKET: profiles S3_ENDPOINT: https://s3.amazonaws.com S3_REGION: us-east-1 # access/secret keys need to be provided if IRSA is not enabled # accesskey: '' # secretkey: '' mongodb: mongoUri: "mongodb://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@cortex-mongodb-replicaset-client.cortex.svc.cluster.local:27017/cortex_services?authSource=admin&replicaSet=rs0" redis: redisUri: 'redis://:$(REDIS_PASSWORD)@cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local:6379' env: REDIS_TYPE: 'sentinel' REDIS_NAME: 'mymaster' REDIS_SENTINELS: '[{"host":"cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local","port":26379},{"host":"cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local","port":26379},{"host":"cortex-redis-node-1.cortex-redis-headless.cortex.svc.cluster.local","port":26379},{"host":"cortex-redis-node-2.cortex-redis-headless.cortex.svc.cluster.local","port":26379}]' REDIS_MODE: 'sentinel' REDIS_SENTINEL_MASTER: 'mymaster' REDIS_SENTINEL_NODES: 'cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local:26379,cortex-redis-node-0.cortex-redis-headless.cortex.svc.cluster.local:26379,cortex-redis-node-1.cortex-redis-headless.cortex.svc.cluster.local:26379,cortex-redis-node-2.cortex-redis-headless.cortex.svc.cluster.local:26379' REDIS_SENTINEL_PASSWORD: '$(REDIS_PASSWORD)'auth: replicaCount: 3accounts: replicas: 3api: replicas: 3catalog: replicas: 3connections: replicas: 3console: replicas: 3models: replicas: 3operator: replicas: 3processorgateway: replicas: 3dex: replicas: 3mongodb-replicaset: replicas: 3minio: enabled: falseredis: sentinel: enabled: true cluster: enabled: true slaveCount: 3cluster-autoscaler: enabled: truedocker-registry: enabled: true secrets: s3: accessKey: '' secretKey: '' storage: s3 s3: region: us-east-1 regionEndpoint: s3.amazonaws.com bucket: docker-registry encrypt: true secure: true persistence: enabled: false