Istio on Cortex

Fabric's gateway for accept incoming requests (Ingress) is the Istio Ingress Gateway, which is part of the Istio service mesh. Its purpose is to provide ingress, routing, authentication, traffic management, and monitoring of Cortex services.

NOTES

• Istio is only recommended in production to be used for service discovery and load balancing.
• The recommended method of installing Istio is with the istioctl utility as there are known networking issues between Dex and external Identity Providers (e.g. PingId or Github) when installing Istio with the Operator.

Istio is required to provide service discovery when scaling Cortex system services and user's daemons. Istio injects sidecars into each pod started in the cortex and cortex-compute namespaces. These sidecars can reserve a significant amount of CPU and memory resources on the cluster if the Istio provided values are used.

Production

Istio components that aren’t required should be disabled, for example: kiali, prometheus, grafana, and tracing. These components can be re-enabled when diagnostic data collection is required.

The default profile provided by the istioctl utility should be used and additional features may be enabled as required.

Requests to Fabric agents via the API require a JWT token or cookie that is passed as part of HTTP Headers.

Authentication is implemented using an authentication proxy and http filters. The http filter forwards requests to the auth proxy validating the JWT headers or cookies. The auth proxy redirects the client to the OIDC provider if an authentication session is required.

Diagnosing issues: telemetry and tracing

In staging or development environments, Istio can be configured to enable tracing and monitoring of pods/services that have the sidecars injected. This allows developers/admins to capture response times, error rates, and other metrics from the cluster during its operation. Istio utilizes Prometheus + Grafana to collect and visualize service mesh metrics. Prometheus is resource-intensive (cpu/ram/disk), so care should be used when deploying it on a cluster.

See the Istio documentation on observability to see how to access the tools bundled with Istio.

Istioctl

The istioctl utility is a streamlined tool to install and manage Istio deployments and can be installed by following the getting started instructions on the Istio docs.

Istio installation config

1. Run the following:

   istioctl install --set profile=default

2. Enable istio sidecard injection on the:

• cortex namespace:

  kubectl label namespace cortex istio-injection=enabled

• cortex-compute namespace:

  kubectl label namespace cortex-compute istio-injection=enabled

Enabling mTLS

In Istio's default profile mutual TLS (mTLS) is enabled and a peer authentication policy is set to PERMISSIVE mode. In order to require communications over mTLS a peer authentication policy may be set up in STRICT mode at the namespace level:

kubectl apply -n cortex -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT
EOF

or for the entire Istio service mesh:

kubectl apply -n istio-system -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT
EOF

The packaged components for MongoDB, Redis, and MinIO do not currently support running in a namespace where mTLS is configured in STRICT mode; therefore, it is recommended to externalize or manage these components independently.

Recommended PeerAuthentication Policies

The following enables strict mTLS across the service mesh but allows exceptions for Spark jobs (SSL can be configured independently via Spark config), MinIO's startup job (if MinIO is enabled), and MongoDB (if not externalized):

kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mongo
  namespace: cortex
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: mongodb
  mtls:
    mode: PERMISSIVE
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: minio-make-bucket
  namespace: cortex
spec:
  selector:
    matchLabels:
      app: minio-job
  mtls:
    mode: PERMISSIVE
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: spark-driver
  namespace: cortex-compute
spec:
  selector:
    matchLabels:
      spark-role: driver
  mtls:
    mode: PERMISSIVE
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: spark-executor
  namespace: cortex-compute
spec:
  selector:
    matchLabels:
      spark-role: executor
  mtls:
    mode: PERMISSIVE
EOF

Additional Istio resources

More detailed instructions are available in the Istio Documentation on further customizing the Istio deployment by setting individual configuration variables or using alternate profiles.