Version: 0.5.6

Istio on Cortex

NOTE: Istio is only recommended in production to be used for service discovery and load balancing.

Istio is required to provide service discovery when scaling Cortex system services and user's daemons. Istio injects sidecars into each pod started in the cortex and cortex-compute namespaces. These sidecars can reserve a significant amount of CPU and memory resources on the cluster if the Istio provided values are used.

Production

Istio components that aren’t required should be disabled, for example: kiali, prometheus, grafana, and tracing. These components can be re-enabled when diagnostic data collection is required.

The minimal profile provided by the istioctl utility should be used and additional features may be enabled as required.

Diagnosing issues: telemetry and tracing

In staging or development environments, Istio can be configured to enable tracing and monitoring of pods/services that have the sidecars injected. This allows developers/admins to capture response times, error rates, and other metrics from the cluster during its operation. Istio utilizes Prometheus + Grafana to collect and visualize service mesh metrics. Prometheus is resource-intensive (cpu/ram/disk), so care should be used when deploying it on a cluster.

See the Istio documentation on observability to see how to access the tools bundled with Istio.

Istioctl

The istioctl utility is a streamlined tool to install and manage Istio deployments and can be installed by following the getting started instructions on the Istio docs.

Istio installation config

Run the following:

istioctl manifest apply --set profile=minimal

Enable istio sidecard injection on the:

cortex namespace:

kubectl label namespace cortex istio-injection=enabled

cortex-compute namespace:

kubectl label namespace cortex-compute istio-injection=enabled

Additional Istio resources

More detailed instructions are available in the Istio Documentation on further customizing the Istio deployment by setting individual configuration variables or using alternate profiles.

Apply Envoy Filter

There is a known issue when using Istio version 1.5.x that causes the Cortex Fabric component versions reported by the /v2/info endpoint to return envoy instead of the service version currently deployed.

In order to resolve this issue, save the following Kubernetes resource to a local file (e.g. file.yaml)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: server-header-passthrough
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH

Apply this resource file to the cortex namespace (or the namespace where Cortex Fabric is installed) with the following command:

kubectl apply -f file.yaml --namespace cortex

#Production

#Diagnosing issues: telemetry and tracing

#Istioctl

#Istio installation config