Version: latest

Release Notes

These Release Notes describes updates released to all SENSA Fabric tools.

Check the Compatibility Matrix for the versions of Cortex tools that correspond with each release.

April 24, 2023 - v6.4.0

DCI v6.4.0, CLI v2.2.2, cortex-python library v6.3.1, Profiles SDK v1.2.x, Cortex Helm Chart v6.4.0

Console

Profile visualizer
- Navigate between Profile links - This allows two profiles to be associated via a relationship using the profile visualizer
Job and Agent Scheduling
- Ability to schedule and trigger jobs using an agent
- Ability to pause, resume, delete, and view the status of job schedules
- Ability to add job schedules from the Agent listing screen
Help icon for accessing product documentation
Usability Enhancements
- Project modal has server-side pagination, sorting, and search
- Profile creation wizard
- Updated the text for the Linked attribute modal as per the UX design
- Update the table to display the error as per the UX design
- Show correct styling for radio buttons
Activation skill info panel
- Download logs for Skills of type daemon

Profiles SDK

PySpark Bindings for Profiles SDK, Ability to run against Databricks clusters and remote Spark clusters
Create/Save profileSchema, dataSource, and connections
Internal service to handle delta bucket connection config for cluster skills
Allow profiles-sdk based jobs to interact with the Cortex managed buckets without the end user needed to know the needed credentials. Currently, the end user needs the Cortex managed bucket credentials and bucket info and needs to set it in the spark-submit configuration. After this change, profiles-sdk can get this info directly from cortex-phoenix when running in a Cortex cluster. Helpers added to profiles-sdk to allow jobs to retrieve internal configuration params for Managed Content Storage
The following scenarios are available as public examples for early access:
- Real-time access pattern to Profiles (query profile)
- Spike to evaluate different options for performing low latency queries for profiles
- Ability to support queries and joins for creating profiles (FAB-3886)

Runtime

Logs with relevant error text are available through the cortex tasks logs api
Agents - Consistency issues with agent state have now been resolved
Skill definitions now support templated daemon.path parameter
Tasks' logs artifacts (stored in Managed Content) now have a default TTL of 90 days. This is configurable through the Helm Chart values
Enhancements to Tasks List API

Campaigns

Deployed Missions now have different monitoring and runtime characteristics[LINK TO NEW DOCS]
Ability to Stop, restart and resume a deployed mission
Ability to view all saved artifacts of a particular mission invocation using activationId

CLI

Add help text to inform user about the new behavior of the soon to be deprecated --query parameter
- Using --query is now equivalent to the previous --json --query <filter expr>
- Filtered tabular results are no longer supported via --query
- The --json param now returns full results in JSON if no value is provided and filtered results if a valid JMESPath expression is provided with --json
Support for Node 16 and above
CLI now uses ESM modules
Workspaces improvements
- Added support for cortex workspaces * --profile <profile>
- Added cortex workspaces publish --skip-push to so you can save a skill locally without pushing it to Docker
- Added Dockerfile path to docker build errors to improve troubleshooting experience
Campaigns Improvements
- Added cortex missions status command to monitor mission status
Removed support for cortex generate

Cortex-Python

Breaking changes to the cortex-python SDK method signatures to simplify usage - More details here

Fabric-ops

This tool is now deprecated in favor of the official Node.js CLI binaries.

Accounts / RBAC

Developer with permission to their own project can list or read resources configured in the shared project via APIs or through CLIs or via Python libraries
As Data-scientist, I should have permissions to READ and WRITE experiments.

Housekeeping

API documentation page update (remove v5 and update v6)
Refactor REST APIs using TypeScript
Add API documentation for Models, Accounts, Secrets, Gateway
Security related upgrades to third party package manifests, and service containers

Examples

Example for synchronous invocation of Agents: Use a Queue in front of processor gateway (add link)
Code examples for Real Time Access to Profiles (add link)

Upgrades

Updates to language runtimes for various Fabric components
Fabric CLI now supports a binary CLI version (Reach out to your support executive for access)

Resolved Issues

Admin is now no longer able to assign a non-existent project to a User or a Role.
cortex content upload no longer fails silently if user tries to upload to a project without necessary permissions
Cortex-Python lib - Project is no longer a required attribute for most Service clients except the root Client
Fixed an issue with Gateway where only the first invoke of an agent assigned to a new node worker thread fails
Improved profile linking behavior in cases where the linked attribute names do not match
Fixed login issues for users with role data-scientist
Fixed an issue with Agent routing in scenarios where the same skill can be called twice in an agent flow
Improved post-login redirection to ensure user always lands on the project provided in the URL fragment or browser cache
Fixed an issue with Streaming Connections where an invalid stream directory resulted in a long running job that did not terminate
Fixed issues with retrieving logs for Daemon Skills
Improved Node.js support for Cortex CLI in Node v16
During Mission Runtime the last batch fails to run when the number of records is less than the configured batch size

Known Issues

Schedules for agents and skills are broken. This will be fixed in 6.4.1
- No known workaround
When Spark-submit is configured to use executors, the GCS key-file is not properly created on the executor(s) filesystem.
- WORKAROUND: This is only an issue when using service account key files, use workload identity in GCS.
Profiles SDK jobs using S3 streaming are not able to be invoked when the auth includes IRSA enabled connections.
- WORKAROUND: Use AWS secret keys for connection authorization rather than IRSA.
In Profiles Calculated Attributes cannot be edited.
- WORKAROUND: To change a Calculated Attribute in Profiles, delete the attribute and create a new attribute with the desired configuration.
KPI values are not being returned at Mission Runtime.
- WORKAROUND: Use the expression var totalCount = count(); rather than var cohortCount = cohort.count() when configuring the KPI.
UpdateProfiles dataset functions (aggregation and union) are not working with multiple executors and delta/current spark-submit configurations.
- WORKAROUND: Use UpdateProfiles locally - When setting a custom spark-submit configuration for the update-profiles service type, the spark.kubernetes.driver.master property must be set to local and spark.kubernetes.executor.*/spark.executor.* properties must not be defined.
Calculated Attributes derived from .csv files are not able to retrieve data for the calculation, causing the Profile ingest to fail.
- WORKAROUND: Avoid configuring Calculated Attributes that pull from CSV files.
When you define a Profile Schema with multiple Data Sources, you cannot have Attributes with duplicate names; this causes the Profile build to fail.
- WORKAROUND: Within a Profile Schema make sure that attribute names coming from different Data Sources are not duplicated.
Skill invoke errors out when payload keys have dots(.).
- WORKAROUND: Make sure that payload keys DO NOT have dots (.).
When a Campaign is imported, a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
- WORKAROUND: Ignore the warning message.
The Mission runtime cycle loses all of the Cohort-Profile data if any of the batches errors out.
- WORKAROUND: If the mission runtime cycle errors, restart the cycle to prevent data loss.
If any Skill or Agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
- WORKAROUND: Check the error log message and manually remove batch-10 data from the file.
If a Project name contains - and _ together or capital letters, then problems getting logs and Skill details occur.
- WORKAROUND: Do not use hyphens, under-scores, or capital letters in Project names.
If an attribute name in a Data Source contains a space, the data ingestion fails.
- WORKAROUND: Do not use spaces in attribute names in Data Sources (and Profiles).
An exception occurs when fetching data (Profiles) where content type is .json in a Filestream Connection.
- WORKAROUND: Do NOT use .json type to define Filestream Connections.
Profile builds will fail if constituent data sources contain only a single record
- WORKAROUND: Ensure you have at least 10 rows of data in each data source that is used in a Profile
Data Source creation from console - Attributes coming from Mongo Connections with documents containing nested types will not be available for selection
- WORKAROUND: Use the Profiles SDK to ingest complex data types coming from MongoDB
Skills that specify the k8sResources attribute in definition can cause the Gateway to crash after multiple invocations
- No known workaround

October 21, 2022 - v6.3.3

DCI v6.3.3, CLI v2.1.1, cortex-python library v6.1.x, Profiles SDK v1.1.0 Cortex Helm Chart v6.3.3

note

SENSA Fabric v6.3.1 and v6.3.2 were internal-only releases.

Profiles SDK

Added functionality to create/save Profile Schema, Data Source, and Connections using the Profiles SDK.
Added an internal service to handle delta bucket connection configuration for System Skills

Fabric Console

Activation Viewer shows the runtime/activation path rather than the agent pattern
Improvements to Mission Simulation:
- Message display
- Behavior following a simulation failure
- Behavior following a browser refresh (state is set to “CANCELLED”)
- Simulation Run button can only be clicked once

Resolved Issues

Campaigns can now be successfully exported.
Agent save in the CLI is no longer failing when updates to the Agent are mistimed.
Profiles update is no longer replacing the Profile record when using the Profiles SDK. Users are now able to fetch original Profiles following an update.
Data Sources using GCS and S3 streaming can now be deleted.
The Activations display for an Agent with multiple incoming services is now accurately showing the single output for each service.
Intermittent Model corruption and Skill router failure is no longer occurring during Mission Runtime in clusters using GCS.

Known Issues

Following long periods of inactivity in the Fabric Console, the user may experience an exception on the first request to the server. The exception depends on the request.
- WORKAROUND: Navigate off of the page in the Console; then return and resubmit the request to get past the error.
When Spark-submit is configured to use executors, the GCS key-file is not properly created on the executor(s) filesystem.
- WORKAROUND: This is only an issue when using service account key files, use workload identity in GCS.
Profiles SDK jobs using S3 streaming are not able to be invoked when the auth includes IRSA enabled connections.
- WORKAROUND: Use AWS secret keys for connection authorization rather than IRSA.
Changes made to Agents during design are not being deployed intermittently. This transient defect causes the original Agent to be invoked rather than an updated version.
- WORKAROUND: This is a transient defect that does not affect production runtime.
In Profiles Calculated Attributes cannot be edited.
- WORKAROUND: To change a Calculated Attribute in Profiles, delete the attribute and create a new attribute with the desired configuration.
KPI values are not being returned at Mission Runtime.
- WORKAROUND: Use the expression var totalCount = count(); rather than var cohortCount = cohort.count() when configuring the KPI.
UpdateProfiles dataset functions (aggregation and union) are not working with multiple executors and delta/current spark-submit configurations.
- WORKAROUND: Use UpdateProfiles locally - When setting a custom spark-submit configuration for the update-profiles service type, the spark.kubernetes.driver.master property must be set to local and spark.kubernetes.executor.*/spark.executor.* properties must not be defined.
Calculated Attributes derived from .csv files are not able to retrieve data for the calculation, causing the Profile ingest to fail.
- WORKAROUND: Avoid configuring Calculated Attributes that pull from CSV files.
During Mission Runtime the last batch fails to run when the number of records is less than the configured batch size.
- WORKAROUND: Make sure that the number of Profiles divides evenly into the configured batch size.
When you define a Profile Schema with multiple Data Sources, you cannot have Attributes with duplicate names; this causes the Profile build to fail.
- WORKAROUND: Within a Profile Schema make sure that attribute names coming from different Data Sources are not duplicated.
Skill invoke errors out when payload keys have dots(.).
- WORKAROUND: Make sure that payload keys DO NOT have dots (.).
When a Campaign is imported, a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
- WORKAROUND: Ignore the warning message.
The Mission runtime cycle loses all of the Cohort-Profile data if any of the batches errors out.
- WORKAROUND: If the mission runtime cycle errors, restart the cycle to prevent data loss.
If any Skill or Agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
- WORKAROUND: Check the error log message and manually remove batch-10 data from the file.
If a Project name contains - and _ together or capital letters, then problems getting logs and Skill details occur.
- WORKAROUND: Do not use hyphens, under-scores, or capital letters in Project names.
If an attribute name in a Data Source contains a space, the data ingestion fails.
- WORKAROUND: Do not use spaces in attribute names in Data Sources (and Profiles).
An exception occurs when fetching data (Profiles) where content type is .json in a Filestream Connection.
- WORKAROUND: Do NOT use .json type to define Filestream Connections.

August 8, 2022 - v6.3.0

DCI v6.3.0, CLI v2.1.0, cortex-python library v6.1.x, Cortex Helm Chart v6.3.0

Fabric CLI

NEW: Workspaces commands for Skill building (Replaces the cortex generate skills command and associated templates)
Agents/Tasks CLI (Task scheduling for jobs)
--list option error message improvements
Delete added for Types and Connections
Validation for Agents has been improved, so users may notice more errors validation occurring for mappings such as: invalid Skill outputs, Skills with no outputs, and invalid Agent outputs.

Fabric Console

New Activations list (searchable) and details pages
Added controls for linking Profiles
Data Source refresh behavior improvements.

Cortex Profile SDK

ANNOUNCING NEW Developer Tool: Cortex Profile SDK

The Cortex Profiles SDK allows for the running and modification of Cortex-empowered Spark jobs. It gives users full control over their data pipeline while ensuring Cortex compatibility by exposing the full breadth of Spark with integrated access to the user's Cortex project.

The Profiles SDK is an extensible dependency injected platform that has process, module, and environment specific configuration. The Cortex Profile SDK includes modules for running locally or in a Cortex cluster with module mocking.

The Profiles SDK provides:

A client library that can be used locally or with ETL tooling to test creation and ingestion of Profiles
The ability to write custom Profile ingestion Skills that can be orchestrated as part of a data pipeline job
Stream and batch processing for ingesting Data Sources and building Profile schemas
The ability to build Profiles using large amounts of data (1 TB)
The ability to incrementally ingest data into Profiles and Data Sources using streaming connections
Parallelization of jobs to handle large data loads
User identity and dataset change tracing
A set of examples showing a number of use cases in cortex-fabric-examples repo

Resolved Issues

If a Profile record is in an error state, the center attributes panel is now continuing to be displayed as expected for configuring Calculated Attributes.
When an Attribute Group name already exists, an error or warning is displayed when attempting to duplicate it in both primary and additional Data Sources.
The following CLI commands are now fully-supported:
- Delete commands for:
  - Models, Experiments, and Runs
  - Secrets
  - Connections

Known Issues

Profiles update is replacing the Profile record when using the Profiles SDK. The error results in users being unable to fetch original Profiles following an update.
Activations display for an Agent with multiple incoming services is not accurately showing the single output for each service.
KPI values are not being updated at mission runtime.
Campaigns cannot be successfully exported.
Profiles SDK is failing to parse when running Spark-submit in cluster mode.
Custom Attribute field does not allow expressions to be written in the Intellicode format that other expression fields allow.
Currently, post review, the count and percentage is calculated only when the train button is clicked; the review count and percentage should be shown once the Refine Plans button is clicked.
Users need a a way to do batch ingest over multiple files with a callback after error/completion. Users need to be able to manually trigger data source ingest on a bucket of files, resume on error, have a callback with completion or error, so further actions can be taken downstream.
When an Attribute Group name already exists, an error or warning is not displayed when attempting to duplicate it in both primary and additional Data Sources.
Agent save in the CLI is failing when updates to the Agent are mistimed.
If the attribute name in a Data Source contains a space, the data ingestion fails.
The Mission runtime cycle loses all of the Cohort-Profile data if any of the batches errors out.
Rewards must be expressed only as float values rather than integers. (e.g. Change rewards from 1 to 0.999 and 0 to 0.001).
When a Campaign is imported a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
When adding calculated attributes to Profiles, the attribute list gets duplicated when there are multiple Data Sources with the same connection. All attributes are displayed multiple times and selected attributes are not removed from the list.
Since the Interventions Log file is a single point of reads/updates, if any of the Skills/Agents fail, the file cannot be reused. The file must be manually updated to remove the error-causing data.
If a Project name contains - and _ together then logs and Skill details cannot be fetched via the CLI.
Special characters in the name of ANY component may cause the Campaign to fail. Users must alter the names in the Hello World tutorial in order to run the Campaign successfully. ADDITIONALLY, Profile Schemas that include addresses and emails cannot be processed by the Skill Router due to the special characters.
Skill invoke errors out when payload keys have dots(.).
If any Skill or Agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
Intermittent Model corruption and Skill router failure is occurring during Mission Runtime in clusters backed by GCS. User is alerted to errors by stagnation of the KPI trending on the Campaign Dashboard and the lack of Profiles being processed (Go here for details). Additionally, Kubernetes errors in K8s are as follows:
- "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 16354: unexpected end of data\n ...","@context":"default"
- "Checksum is inconsistent, file is possibly corrupted.', 'locations': [{'line': 2, 'column': 3}], 'path': ['interventionsForProfile'],"
Data Sources using GCS and S3 streaming cannot be deleted.
During Mission Runtime the last batch fails to run when the number of records is less than the configure batch size.

June 21, 2022 - v6.2.2

DCI v6.2.2, CLI v2.0.13, cortex-python library v6.1.x, Cortex Helm Chart v6.2.2

Security and functional enhancements completed.

Resolved Issues

Data Source ingestion is no longer being limited when running Spark; large job batches are succeeding.

Known Issues

When an Attribute Group name already exists, an error or warning is not displayed when attempting to duplicate it in both primary and additional Data Sources.
Agent save in the CLI is failing when updates to the Agent are mistimed.
If the attribute name in a data source contains a space, the data ingestion fails.
The mission runtime cycle loses all of the cohort-profile data if any of the batches errors out.
Rewards must be expressed only as float values rather than integers. (e.g. Change rewards from 1 to 0.999 and 0 to 0.001).
When a Campaign is imported a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
When adding calculated attributes to Profiles, the attribute list gets duplicated when there are multiple Data Sources with the same connection. All attributes are displayed multiple times and selected attributes are not removed from the list.
Since the Interventions Log file is a single point of reads/updates, if any of the Skills/Agents fail, the file cannot be reused. The file must be manually updated to remove the error-causing data.
If a project contains - and _ together in the project name then problems getting logs and skill details occur.
If a Profile record is in an error state, the center attributes panel is empty when you attempt to configure Calculated Attributes.
Special characters in the name of ANY component may cause the Campaign to fail. Users must alter the names in the Hello World tutorial in order to run the Campaign successfully. ADDITIONALLY, Profile Schemas that include addresses and emails cannot be processed by the Skill Router due to the special characters.
Skill invoke errors out when payload keys have dots(.)
If any skill or agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
The following CLI commands (listed in the CLI for this release) will NOT be fully-supported until the subsequent release:
- Delete commands for:
  - Models, Experiments, and Runs
  - Secrets
  - Connections
Intermittent Model corruption and Skill router failure is occurring during Mission Runtime in clusters backed by GCS. User is alerted to errors by stagnation of the KPI trending on the Campaign Dashboard and the lack of Profiles being processed (Go here for details). Additionally, Kubernetes errors in K8s are as follows:
- "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 16354: unexpected end of data\n ...","@context":"default"
- "Checksum is inconsistent, file is possibly corrupted.', 'locations': [{'line': 2, 'column': 3}], 'path': ['interventionsForProfile'],"
Data Sources using GCS and S3 streaming cannot be deleted.
During Mission Runtime the last batch fails to run when the number of records is less than the configure batch size.

May 21, 2022 - v6.2.1

DCI v6.2.1, CLI v2.0.13, cortex-python library v6.1.x, Cortex Helm Chart v6.2.1

Defects and vulnerabilities have been addressed.
Agent returns response synchronously if sync flag is set to “true” and contain a singleton skill.
CLI: List filters have been enhanced.

Resolved Issues 6.2.1

During Mission Runtime, jobs no longer appear to remain in the PENDING state.
In the CLI the --filter tag now works in Agents list-activations.

Known Issues 6.2.1

Data Source ingestion is being limited when running Spark, so that large job batches are returning an Out-of-Memory error.
Agent save in the CLI is failing when updates to the Agent are mistimed.
If the attribute name in a data source contains a space, the data ingestion fails.
The mission runtime cycle loses all of the cohort-profile data if any of the batches errors out.
Rewards must be expressed only as float values rather than integers. (e.g. Change rewards from 1 to 0.999 and 0 to 0.001).
When a Campaign is imported a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
When adding calculated attributes to Profiles, the attribute list gets duplicated when there are multiple Data Sources with the same connection. All attributes are displayed multiple times and selected attributes are not removed from the list.
Since the Interventions Log file is a single point of reads/updates, if any of the Skills/Agents fail, the file cannot be reused. The file must be manually updated to remove the error-causing data.
If a project contains - and _ together in the project name then problems getting logs and skill details occur.
If a Profile record is in an error state, the center attributes panel is empty when you attempt to configure Calculated Attributes.
Special characters in the name of ANY component may cause the Campaign to fail. Users must alter the names in the Hello World tutorial in order to run the Campaign successfully. ADDITIONALLY, Profile Schemas that include addresses and emails cannot be processed by the Skill Router due to the special characters.
Skill invoke errors out when payload keys have dots(.)
If any skill or agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
The following CLI commands (listed in the CLI for this release) will NOT be fully-supported until the subsequent release:
- Delete commands for:
  - Models, Experiments, and Runs
  - Secrets
  - Connections
Intermittent Model corruption and Skill router failure is occurring during Mission Runtime in clusters backed by GCS. User is alerted to errors by stagnation of the KPI trending on the Campaign Dashboard and the lack of Profiles being processed (Go here for details). Additionally, Kubernetes errors in K8s are as follows:
- "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 16354: unexpected end of data\n ...","@context":"default"
- "Checksum is inconsistent, file is possibly corrupted.', 'locations': [{'line': 2, 'column': 3}], 'path': ['interventionsForProfile'],"
Data Sources using GCS and S3 streaming cannot be deleted.
During Mission Runtime the last batch fails to run when the number of records is less than the configure batch size.

May 6, 2022 - v6.2.0

DCI v6.2.0, CLI v2.0.12, cortex-python library v6.1.x, Cortex Helm Chart v6.2.0

Fabric developers can create and update a GCS or GCS Streaming connection from the Cortex Console.
Fabric developers can save, list, delete, and describe a GCS or GCS Streaming Connection from the Cortex CLI. (Note: CLI is released separately)
Fabric developers can create and ingest a data source using a new GCS connection.
Fabric developers can create a new Profile schema and ingest data from a Data Source created using a GCS Connection or GCS Streaming Connection.
Files can be uploaded to and read from a GCS Connection.
Data Source ingestion from files has been made more memory efficient.
Cortex Managed Content can now be backed by GCS.

Resolved Issues 6.2.0

The Mission names exceeding 63 characters are now supported in Online Learner.
GKE cortex-api service is now honoring the REDIS_HOST env variables override.
You can now edit a Profile Schemas in the Fabric Console.

Known Issues 6.2.0

Agent save in the CLI is failing when updates to the Agent are mistimed.
If the attribute name in a data source contains a space, the data ingestion fails.
The mission runtime cycle loses all of the cohort-profile data if any of the batches errors out.
Rewards must be expressed only as float values rather than integers. (e.g. Change rewards from 1 to 0.999 and 0 to 0.001).
When a Campaign is imported a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
When adding calculated attributes to Profiles, the attribute list gets duplicated when there are multiple Data Sources with the same connection. All attributes are displayed multiple times and selected attributes are not removed from the list.
Since the Interventions Log file is a single point of reads/updates, if any of the Skills/Agents fail, the file cannot be reused. The file must be manually updated to remove the error-causing data.
If a project contains - and _ together in the project name then problems getting logs and skill details occur.
If a Profile record is in an error state, the center attributes panel is empty when you attempt to configure Calculated Attributes.
Special characters in the name of ANY component may cause the Campaign to fail. Users must alter the names in the Hello World tutorial in order to run the Campaign successfully. ADDITIONALLY, Profile Schemas that include addresses and emails cannot be processed by the Skill Router due to the special characters.
Skill invoke errors out when payload keys have dots(.)
If any skill or agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
The following CLI commands (listed in the CLI for this release) will NOT be fully-supported until the subsequent release:
- Delete commands for:
  - Models, Experiments, and Runs
  - Secrets
  - Connections
During Mission Runtime, jobs appear to remain in the PENDING state. To determine when the jobs have completed view the Kubernetes pod logs.
Intermittent Model corruption and Skill router failure is occurring during Mission Runtime in clusters backed by GCS. User is alerted to errors by stagnation of the KPI trending on the Campaign Dashboard and the lack of Profiles being processed (Go here for details). Additionally, Kubernetes errors in K8s are as follows:
- "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 16354: unexpected end of data\n ...","@context":"default"
- "Checksum is inconsistent, file is possibly corrupted.', 'locations': [{'line': 2, 'column': 3}], 'path': ['interventionsForProfile'],"
Data Sources using GCS and S3 streaming cannot be deleted.
During Mission Runtime the last batch fails to run when the number of records is less than the configure batch size.

March 30, 2022 - v6.1.1

DCI v6.1.1, CLI v2.0.8, cortex-python library v6.1.x, Cortex Helm Chart v6.1.1

note

Version 6.1.0 has been omitted from these release notes as it was an initial version of the content in this official release.

Backend

Support for sending a response message for each record in a job. (The default is to send a single response message when the job completes.)
Support for Kafka for streaming Agent services has been added.
Long running queries against Profiles can now be supported.
Managed Content handler is now part of discovery_connectors.

CLI

Commands have been added to list, create, view details, and delete Sessions.
Commands have been added to publish and unpublish Models.
A dependency check is run when you delete Skills, Actions, and Agents from the CLI.
cortex generate skill file output has been modified.
--scaleCount and --podspec can now be defined for cortex skills save.
list, get or describe, delete and logs commands have been added to CLI to manage Kubernetes Tasks (Jobs).
The list command's response for all Fabric assets has been enhanced to provide more information.
correlationId can be added to Skill's payload.

Console

Controls have been added to publish and unpublish models from the kebab menu in Models list view.
Controls have been added to deploy and undeploy Missions from the Workflow panel (at the bottom right) in the Mission Control view and from the Mission Card kebab menu in the Missions list view.
In Skill Builder the properties for model-artifact, model-id are now editable.
Added statuses for the Refining Plans process in Mission planning.
Added statuses for jobs. Statuses now include: SUBMITTED, QUEUED, ACTIVE, COMPLETE, FAILED/ERROR.
Agent Composer now supports arrays for Skill properties.
Key metrics are now displayed in the Workflow panel for Mission Simulation and Plan Refinement.
You can now add tags to Profile Attributes for grouping using the Profile Viewer in Console.
Improved in app guidance and messaging added to Campaign Design in the Console.

SENSA Fabric Examples

Windows scripts examples have been added for Intervention Skill deployment
Flu-shot intervention Skills and associated synthetic data has been moved into the cortex-fabric-examples repo.

Known Issues Resolved

Connections are now exported with Skills.
You must have a Goal saved in order to configure a Mission.
Dramatic resource consumption during mission runtime has been resolved.
After you rebuild a Data Source that has been updated, the Profile rebuild now succeeds, as expected.
Profile Data is saved to a Profiles (parquet) file when the Cohort Manager is called for the first time. As each batch is called, the profiles in the batch are deleted from that file. If any of the batch-iterations errors-out, the profiles data for the batch is no longer lost.
When creating an S3 connection, an error is displayed when you attempt to save without entering all of the required details.
All available Interventions are being randomly suggested for all Cohort members in the batch.
When you define a Profile Schema and attribute group names are duplicated, the second group name is ignored.

Known Issues

The Online Learner pod fails when the Mission name exceeds 63 characters. (NOTE: Mission Name is a concatenation of dataSourceName, profileName, and missionName).
Attempting to edit a Profile Schemas in the Console results in a browser crash.
In GKE cortex-api service is not honoring the REDIS_HOST env variable override.
In the CLI the --filter tag does not work in Agents (list-activations).
Agent save in the CLI is failing when updates to the Agent are mistimed.
If the attribute name in a data source contains a space, the data ingestion fails.
The mission runtime cycle loses all of the cohort-profile data if any of the batches errors out.
Rewards must be expressed only as float values rather than integers. (e.g. Change rewards from 1 to 0.999 and 0 to 0.001).
When a Campaign is imported a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
When adding calculated attributes to Profiles, the attribute list gets duplicated when there are multiple Data Sources with the same connection. All attributes are displayed multiple times and selected attributes are not removed from the list.
Since the Interventions Log file is a single point of reads/updates, if any of the Skills/Agents fail, the file cannot be reused. The file must be manually updated to remove the error-causing data.
If a project contains - and _ together in the project name then problems getting logs and skill details occur.
If a Profile record is in an error state, the center attributes panel is empty when you attempt to configure Calculated Attributes.
Special characters in the name of ANY component may cause the Campaign to fail. Users must alter the names in the Hello World tutorial in order to run the Campaign successfully. ADDITIONALLY, Profile Schemas that include addresses and emails cannot be processed by the Skill Router due to the special characters.
Skill invoke errors out when payload keys have dots(.)
If any skill or agent in the Mission runtime cycle fails, the intervention_logging.parquet file becomes corrupted.
The following CLI commands (listed in the CLI for this release) will NOT be fully-supported until the subsequent release:
- Delete commands for:
  - Models, Experiments, and Runs
  - Secrets
  - Connections

November 29, 2021 - v6.0.10

DCI v6.0.10, CLI v2.0.7, cortex-python library v6.1.x, Cortex Helm Chart v0.6.10

The features for this release were focused on Helm and Platform changes. For details view the Charts Changelog.

Documentation

Define Skills page has been updated to reflect new files and processes.
“Add Users” section has been added to RBAC Administration Guide.
Campaign Use Case Examples have been added to the docs to help users understand Campaign development and execution.
The Onboarding Guide has been added to help users get started using Fabric.

Known Issues Resolved

The following known issues have been resolved in this release:

Missions can now be deleted.
Status of the Online Learner agent (exposed only in K8s) moves from PENDING to COMPLETED after the process is completed.
In Profiles you can now add a Calculated Attribute that references another just created Calculated Attribute without refreshing the browser.
Podspec can now be applied to jobs using the CLI.
In the CLI the Data Scientist Role is now able to access a project assigned to that role.
Goals now display a warning when a user enters an invalid KPI expression and they are able to edit expressions that have been saved.

Known Issues (in this release)

Execution of the Mission runtime may result in dramatic resource consumption, increasing platform fees.
If a Profile record is in an error state, the center attributes panel is empty when you attempt to configure Calculated Attributes.
No error message is displayed when a user attempts to save a Mission without selecting a Goal. Since the Goal selection is required, saving the Mission fails, but no message is displayed.
Special characters in the name of ANY component may cause the Campaign to fail. Users must alter the names in the Hello World tutorial in order to run the Campaign successfully.
The -k or --k8s argument only works with cortex actions deploy; It is not supported for cortex skills deploy.
After you rebuild a Data Source that has been updated, the Profile rebuild fails.

cortex skills invoke errors out when the payload includes keys with dots, for example:

{
  "payload": {
    "repsonse": {
      "KEY_CHK_DCN_NBR": "U4389D",
      "TOP1_1.5_version": "45"
    }
  }
}

Oct 13, 2021 Release version 6.0.9

DCI v6.0.9, CLI v2.0.5 and v2.0.6, cortex-python library v6.0.3, Cortex Helm Chart v0.6.9

Backend

Support added for k8s Resources:
- Horizontal Pod Autoscaler (HPA)
- Pod Disruption Budget (PDB)
- Istio Virtual Service
- Istio Destination Rule
- Istio Traffic Management:
  - Retries
  - Timeouts
  - Circuit breakers
A compile check runs automatically when you deploy a Campaign and warning messages are displayed for dependencies such as Skills for interventions, invalid expressions in Interventions.
- Script expressions entered in the UI are validated during the compile check.
The Impact Assessment tool now supports additional dependency tracking.
Support added for Cortex JDBC and CData Connections.
An API is now available to trace model metrics in the Mission runtime cycle.

CLI

Models Registration now includes support for Model Mode and Type.
Save actions options now include -k', '--k8sResource which enables additional Kubernetes resources to be deployed and owned by the Skill.
JSON and YAML are now accepted by the CLI without specifying the -y option.
cortex deploy <agent, skill, campaign, profile> <name> added to support export of bill-of-materials for promotion to a different environment CI/CD integration.

Fabric Console

Profile Viewer now displays attribute types.
Error messages are displayed when issues occur during Data Source and Profile data ingestion.
Agent properties and headers are retained when an Agent is saved from the UI.
Intervention configuration now includes a Wait Time Field which specifies the feedback collection period.
Models Registration now includes fields for model Mode and Type.
During Skill creation, only published Models can be selected.
Model Status is now displayed.
Properties and Parameters added to Skills during Skill creation are automatically validated.
A ReadOnly view is available while editing a Mission so original selections can be reviewed in context.
Model Experiments can now be managed in Fabric Console.
A warning message had been added to the simulation specification page when the mission is not in EDITING mode.
Mission status is now displayed on the Campaign summary page in the Missions list box.
Under the Simulation History tab a reach bar displays the completion of the historical simulations.
New Connection types, Cortex JDBC and CData JDBC have been added.
Improvements to Agent Composer developer tray:
- Both the "latest test case" tray and the "trace output" are expandable.
- The playback timeline zoom slider in the "latest test case" tray is always displayed.
Option to deploy a campaign from the Campaign Card or Campaign landing page.

SENSA Fabric Examples

JDBC CData Connection Skill template
Batch prediction - Mongo dependencies added
Pymongo connection example

Python Library

Online Learner - support for:
- Agent Orchestration
- Feedback and Rewards
- Model metrics
- Log components added
- Runtime repository
- Using public databases

Known Issues

If a project contains - and _ together in the project name then problems getting logs and skill details occur.
In the CLI the Data Scientist Role is unable to access a project assigned to that role.
Missions cannot be deleted at this time.
Status of the Online Learner agent (exposed only in K8s) remains PENDING even after the process is completed.
Goals cannot be saved if a user enters an invalid expression and attempts to edit it.
Connections are not automatically exported for Skills. You must reconfigure them in your new environment.
In Profiles to add a Calculated Attribute that references another (just created Calculated Attribute, you must rebuild the Profile before the second Attribute can be added. Calculated Attributes are not pulled into the Profile until after the Profile rebuilds when it is saved.
Since Interventions Log file is a single point of reads/updates, if any of the skills/agents fail, the file cannot be reused. The file must be manually updated to remove the error-causing data.
As the Interventions Log file-size grows, the system runs more slowly.
Profile Data is saved to a Profiles(parquet) file when the Cohort Manager is called for the first time. As each batch is called, the profiles in the batch are deleted from that file. If any of the batch-iterations errors-out, the profiles data for the batch is lost.
Rewards must be expressed only as float values rather than integers. (e.g. Change rewards from 1 to 0.999 and 0 to 0.001).
Profile Schemas that include addresses and emails cannot be processed by the Skill Router due to the special characters.
When adding calculated attributes to Profiles, the attribute list gets duplicated when there are multiple Data Sources with the same connection. All attributes are displayed multiple times and selected attributes are not taken removed from the list.
When a campaign is imported a warning about missing secrets is displayed regardless of whether or not the Campaign uses secrets.
Only one intervention is being suggested for all pCohort members in the batch.

August 19, 2021 Release v6.0.8

DCI app version: 6.0.8, Cortex Helm Chart version 0.6.8, CLI 2.0.5, cortex-python library v6.0.3

Backend

System Skills:
- Spark submit-job Skill for running Spark jobs
- Agent Invoke (for Agent sync)
Agent sync feature (Agents invoking Agents synchronously)
Impact Assessment enhancements for identifying impacts to connections, catalog components, and models
Impact Assessment API methods for displaying impact assessment dependency tree for Agents and Campaigns
podSpec field added to action definition for Skills

CLI Commands

Deploy campaign artifacts via CI/CD
Undeploy campaigns
Validate mission status prior to deployment
Agent activation query returns correlationId, which provides a method by which Agents can synchronously invoke another Agent for deploying more sophisticated AI solutions

Console

Console alerts have been improved and toasts have been added
Copy to clipboard functionality has been added for links to Impact Assessments
Agents List and Skills List pages have been amended to add details
Delete verifications have been added for all resource types in Console
Podspec (charts override) option has been added to Skill builder
Profile Visualizer toolset has been added to Console, so you can now drill down into profiles to explore attributes
Model ID is now displayed when you register a model in Console
The Intervention Lifetime field has been added to Intervention configuration, so you can specify how long the intervention should run
Table pagination has been added to the Impact Assessment list view.

SENSA Fabric Examples (GitHub repo)

Python Lib Session example
A daemon Skill template for serving REST API prediction requests
A Skill template for batch predictions

Python Library

Support added model CRUD activities
Support added for creating experiments using models

Known Issues

AgentDef loses properties when saved from UI.
When you save an Agent, the property validValues: [] is automatically introduced, even when it is not supplied in body. However, you cannot save an Agent if you manually add the property to the Agent or if the Agent has been previously saved (and the property was created).
Podspec is not being applied to Skills that have been previously deployed. To pick up podspec config undeploy and redeploy the Skill.
In Fabric Console Agents with jobs are not displaying time or trace outputs. Invoke using the CLI to display job responses.

June 30, 2021 Release v6.0.7

DCI app version: 6.0.7, Cortex Helm Chart version 0.6.7, CLI 2.0.4, cortex-python library v6.0.3

info

v0.6.6 was a hot-fix release.

Backend

New Cortex CLI commands are now available for working with Campaigns, Missions, Assessments, and Models.
Logging and Tracing support has been added to the Cortex Agents.
Runtime statuses for Agents, Actions, and Campaigns have been updated.
Cortex Python Library has been updated to support Cortex v6 activities
Support for bulk-batch processing has been added.
Scaling for simulations requiring large numbers of records has been improved
Performance improvements for loading Profile and feature catalog content have been made.
HA/DR readiness work has been completed.
Changes to Cortex components such as Profile, Attributes, Missions, Agents, Skills can now be assessed for impact
Data Scientists can register new ML Models
Developers can now use ML Models to run bulk batch predictions
Missions can be deployed and executed as an Agent
Profile data can now be ingested using streaming API
Connections, especially S3 can now be created using AWS IAM Roles for Service Accounts (IRSA)
New quick start examples are added to the SENSA Fabric examples
New GQL Endpoints have been added to track mission interventions and feedback
New type of Skills have been added to support common agent invocation patterns

Console

Project button moved to left panel; clicking it exposes a modal where Projects are selected and managed.
The Settings page that contains the personal access token has been moved to the top right under the gear icon.
Assessments page has been added for conducting and reviewing impact analysis for component management.
A new Connections type option has been added to support data streaming.
A new System Skill has been added to support Agents where Skill outputs are merged.
The Campaign UI now support Mission Deployment and feedback.
Model Registration support has been added to the Console.
Skill building UI has been added to the Console.
Impact Assessments reports can be generated using the new Assessment UI

May 11, 2021 Release v6.0.5

DCI app version: 6.0.5, Cortex Helm Chart version 0.6.5, CLI 2.0.3, cortex-python library v6.0.2

info

v6.0.4 was a hot-fix release.

Backend

Removed unnecessary fields from the get activation cli command response
- NOTE: For backward compatibility and to view all properties, parameters, inputs, and outputs append the cli command with --verbose.
Removed system fields from describe agent/skill cli command response
- NOTE: For backward compatibility and to view all properties, parameters, inputs, and outputs append the cli command with --verbose.
Added support for "Shared" catalog items to enable system-Skills and sharing catalog items across Projects.
Added an agent-invoke shared skill, which enables Agents to invoke other Agents.
- KNOWN ISSUE: activationId is currently only propagated to the first Agent invoked; subsequent Agents are not assigned a unique activationoId .
ThesessionId returned during Agent/skills invoke now defaults to the activationId to allow for greater traceability.
Skill deploy/undeploy REST APIs and CLI commands have been added to Fabric so that Skills can be undeployed and redeployed to Kubernetes without modifying the Skill definition in the catalog.
Profiles can now be deleted in Fabric.
When you remove attributes or custom attributes from a Profile schema, the Profile features are automatically updated (rebuilt).
In the CLI you can now filter the activation list by timestamp and status.

Console

The Cohort Group page has been added to Console to display a sample of profile attributes and statistics for the group.
Cohort Groups can now be deleted from the Cohort Group card displayed on the Campaign Cohort page.
Upgraded Node.JS and third party dependencies to resolve CVEs.
A new property, simulationType, has been added to intervention plans to distinguish between plans resulting from the initial simulation and plans that have been regenerated following the refinement process.
Updated scrolling behavior on list pages in the Campaigns section of Console.
Custom Attributes setup in the Profile Schema wizard now prevents the creation of duplicate attributes.
Console breadcrumb bar remains visible when scrolling down pages.
The Simulated Attributes setup in the Mission creation wizard looks and behaves similarly to the Custom Attributes step in the Profile Schema wizard.
Simulation status and errors have been added to the History tab on the Mission page.
All component creation dialog name generators (e.g. Add Project, add Profile Schema) have been updated to reflect new backend constraints that require names to be alphanumeric, beginning with a letter and ending with a letter or number. In between dashes and underscores are allowed; no other special characters can be used.
A new Profiles page has been added to the profile details tab-set; it is used to display a sample of profile attributes and statistics.

Documentation

Bug Fixes

The secrets API is now able to manage trailing slashes in configuration/keys.
Custom Profile attributes are now scoped to the profile schemas for which they were created.
When you refine plans, the selection of a rank of less than zero is no longer available.
When you view plans for a mission, only attributes associated with a simulation are displayed as grouping selection options.
Interventions are now being displayed for Missions in the DEPLOYED state.
In the Profile Schema wizard when you navigate to another page, you are only prompted about unsaved changes if you have made changes.
The Profile Schema wizard no longer saves unwanted changes.
The "Add Project" button is only displayed for users with project creation permissions

April 7, 2021 Release v6.0.3

DCI app version: 6.0.3, Cortex Helm Chart version 0.6.3, CLI 2.0.2, cortex-python library v6.0.2

Backend

To enhance security secret values are no longer displayed in the Console or CLI.
A User column has been added to Profile and Data Source create/update log tables for traceability.
Improvements in error response and handling have been added to mission refinement.
Improved error handling and messaging has been added to the Cortex Catalog REST API.
Agents and Skills properties now use the #SECURE.<secret-key> marker to reference secret values.
Skill output names can now be specified in responses and referenced in Agents for more flexible orchestration of Skills in an Agent.
Type parameter names (for mapping in Agents) must be unique within the type resource. (e.g a schema cannot have two parameters called "name")
Reenabled the Sessions REST API (in Gateway section).

Console

Simulation History tab has been added to Missions for selecting and displaying previous simulations.
In mission planning Cost and Probability sliders now default to 0.01 (rather than 0).
A confirmation/warning modal has been added when navigating away with unsaved changes in Cohort, Data Source, Goal, Mission, and Profile Schema editors.
A new page displays a sampled list of profiles for a Cohort Group.
Support for deleting Missions, Campaign Goals, and Campaign Cohorts has been added.
Secrets selection has been added back to Agent Composer in the Skill's Properties panel.

Documentation

Bug Fixes

Project creation dialog now prevents creation of a new Project with an existing name.
"N/A" is now displayed for any missing/unknown data on Campaign KPI cards.
Fixed incorrect usage/persistence of create/delete/update for Custom and Bucket attributes in Profile Schema wizard.
Priority in Refine Plan no longer accepts values less than zero.
Profile Schema Library loads consistently.
When a Connection is created or edited, dependent properties are displayed when the parent property is configured.
Improvements made to error display consistency and placement.
"Test Connection" button has been removed since that functionality is not used in Fabric v6.

March 4, 2021 - Release v6.0.2

DCI app version: 6.0.2, Cortex Helm Chart version 0.6.2, CLI 2.0.1, cortex-python library v6.0.2

Backend

Profile queries for mission plans have been added.
Support has been added for multiple simulations running at the same time in a cluster.
Standardization around naming conventions has been added. Namespace requirement has been removed from Skills, Agents, and connections
Security improvements have been made.
Node version in the container has been updated to 12.20.1 and some third-party dependencies used to resolve CVEs (Common Vulnerabilities and Exposures) have been updated.

Console

Support for editing Data Sources has been added.
Support for editing campaign goals has been added.
Access to change log tabs for Data Sources and profile schemas has been added.
Mission Intervention Pre-Conditions are now scoped across the mission so they can be selected for multiple interventions.
Support has been added for editing simulated attributes from inside a mission (previously only available when creating/editing mission metadata).
Prompts have been added that identify component creation prerequisites throughout the Fabric Console.
Added support for configuring custom key-value parameters to Connections.
Improved page load time.
When a simulation starts, the right-side workflow panel now auto-closes, so the progress bar and full animation are displayed.
After a simulation runs and generates plans, the two grouping selectors in Mission Control now constrain which attributes are available for plan-group selection. Only attributes that are categoricals and strings can be selected; booleans can no longer be selected.
The View Plans popup has added "Reached Goal" percentage.

Documentation

Content about using files as commands with the CLI (especially recommended for Windows users) has been added.
Building Skills section has been added.
All content around Agent lifecycle and Agent building has been rewritten to reflect v6 changes.

Bug fixes

Resolved timeout issues.
Resolved simulation issues.
Fixed icon rendering defect in Console (e.g. lock icon displayed when agents were not locked).
Resolved issue that caused the workflow buttons to disappear after running a simulation.
Resolved issue where simulation animations could fail to render when intervention titles contained unexpected characters.

Known issues

The following issues are related to "namespace" deprecation:

Creating a data source with "/" in the name errors out.
Creating or updating an Agent with "/" in the name may fail upon saving; no error message is displayed.
After your first Campaign simulation run, you may encounter failures on subsequent runs.
When updating a Profile custom attribute to "bucketed", a known issue in the UI causes the attribute to be dropped unless you click the Add Bucket button and supply a bucket name. Alternatively, you can delete the original custom attribute and create a new bucketed one with the same name.

December 16, 2020 - Release v6.0.1

DCI app version: 6.0.1, Cortex Helm Chart version 0.6.1, CLI 2.0.1, cortex-python library v6.0.1

Features

Import/Export Campaigns using the CLI to share Campaigns between Projects
API service added to Cortex system-health check
CogU - Campaign Building Tutorials (Cortex V6 -Tutorials)
- Introduction to Campaign Building
- Prepare Campaign Prerequisites
- Build a Campaign
- Flu Shot Tutorial Data Dictionary (provides data to enter in conjunction with tutorials to complete the Flu Shot use case)

Dependency releases

Glossary has been updated with additional v6 concepts
CLI reference guide has been updated
Python-lib reference guide major version release (This does not include python-profiles and python-builders libraries)
caution
If you are using Cortex version 5.x.x, do NOT update your Python Library version to 6.0.x. The new version contains breaking changes and should only be used with Cortex v6.

Improvements

Console simulation performance is faster
Console user interface changes to provide better user experience
Bug fixes and other general improvements

Work in progress

Great things are happening in Cortex V6, but all great things take time. Please be patient while we complete work on:

Campaign deployment
Integration of Profile Explorer
Runtime logs and monitoring tools
TTL and scaling
Jobs and triggers
Documentation around agent and skill development
Debug - Use a log aggregator of your choice with Kubernetes to help with this.

These are some specific known issues to be aware of:

Page refresh during simulation may cause problems in the Console UI.
When building Skills, you must save the action that the Skill references BEFORE you save the Skill. If you attempt to save the Skill first, an error is returned when you invoke the skill.
When you attempt to stop a simulation from the Console, it may not stop running.
For Agents, Skills, and actions make sure that the combination of Namespace/Name is less than 20 characters. This will be enforced in the API and UI in a future release, but be aware that you may encounter issues with long names.

December 7, 2020 - Release v6.0.0

DCI app version: 6.0.0, Cortex Helm Chart version 0.6.0, CLI 2.0.0

Features

Welcome to the first release of SENSA Fabric V6. We are excited to offer you many new features including:

In SENSA Fabric Console you can now:
- Build and manage Campaigns
- Plan missions
- Simulate, and refine Missions
- Configure and ingest Data Sources
- Build profile schemas & profiles from multiple Data Sources
- Select projects based on user permissions
- Download personal access tokens for client-side access

Improvements

Agent instances are no longer used; Snapshots now provide the bill-of-materials required to deploy and invoke agents
Skills and Agents (not actions) are invoked at runtime
Agents are built, edited, and run in Console's Agent Composer (These functions have been re-added.)
A number of fixes and enhancements from the Preview releases

Work in progress

Great things are happening in Cortex V6, but all great things take time. Please be patient while we complete work on:

Campaign deployment and reinforcement learning
Integration of Profile Explorer
Runtime logs and monitoring tools
Updating the Python Library to fully support V6
TTL and scaling
Jobs triggers
Removing references to datasets in Agent Composer
Documentation
Debug - Use a log aggregator of your choice with Kubernetes to help with this.

These are some specific known issues to be aware of:

For agents, skills, and actions make sure that the combination of Namespace/Name is less than 20 characters. This will be enforced in the API and UI in a future release, but be aware that you may encounter issues with long names.
Agent Composer is still undergoing changes, and you may run into issues. If you encounter problems, the CLI is available as a fallback for working with agents.

November 25, 2020 - Preview release v6.0.0-a.2

DCI app version: 6.0.0-a.2, Cortex Helm Chart version 0.6.0-a.2, CLI 2.0.0-a.2

This is the second SENSA Fabric v6 Preview Release. Progress is messy so pardon our dust. Agent Composer is still undergoing updates to work with v6. You may encounter issues when creating or editing Agents in the Console. The formal v6 release is right around the corner.

Deprecations in the release:

Datasets - These are being deprecated and will be fully replaced by Data Sources soon; however, remnants of datasets remain in Agent Composer and the CLI.
Triggers - (as a component of a dataset triggers are deprecated) A new way of setting triggers is coming soon
Connection Types - Many of the v5 connection types have been deprecated. If you want to use a Cdata connection, contact your CognitiveScale representative
Actions (Jobs and Daemons) can no longer be invoked outside of a Skill or Agent.

This preview contains:

Vault secrets engine (cortex secrets)
- Cortex clusters can be configured to store secrets in your preferred vault provider (tested with Hashicorp)
- Secrets may be saved using the CLI to support secure access to external resources (e.g. automated pipelines deploying assets from one environment to another)
Connections changes
- v5 connections are deprecated
- New Connection types for v6 include: local file, Mongo, S3, and Hive (Cdata connections available upon request)
- Secrets can now be used in any secret-enabled connection fields, like passwords
- Fabric Console: Connections' secret-enabled fields display a dropdown where users select a predefined secret
Authorization
- By default, the following roles are defined out of the box
  - Administrator
  - AI Developer
  - Data Engineer
  - Business User
- Role-based access control in the Console ensures that roles can only access pages they have access to based on project grants
Agents are now self-contained deployments using Kubernetes controllers, which removes invocation bottlenecks.
Skills:
- New Skill type added (External API) to existing types (daemon and job) to support consuming external REST APIs without having to write a daemon.
Additional Docs: (view docs on release features at the links)
- Services List has been updated for V6 Services
- Glossary has been updated with key v6 concepts
- CLI reference guide has been updated
- Python-lib reference guide has been updated
Changes to Python Libraries including:
- New authentication methods that work with v6 authorization and APIs.
- Deprecated and new functionality

note

Several changes have been made to cortex-python library to better work with Cortex v6 and to focus more on methods that support skill development and running experiments. Several methods have been removed related to agents, pipelines, datasets, and other areas. More updates are coming soon.

cortex-python-builders and cortex-python-profile libraries are not yet available for Cortex v6.

October 30, 2020 - Preview release v6.0.0-a.1

DCI app version: 6.0.0-a.1, Cortex Helm Chart version 0.6.0-a.1, CLI 2.0.0-a.1

Welcome to SENSA Fabric v6 Preview Release. We are in the process of making significant improvements to Cortex. Our Preview release includes:

A new single-sign-on(SSO) authentication and authorization system that configures your own identity provider such as Ping Identity or Active Directory to work seamlessly with Cortex (from Helm), so you no longer have to manage users in the Cortex system.
The ability to map the groups in your identity and authorization provider to Cortex roles (developer or administrator in this release)
A new, more secure method to access the CLI by obtaining a security token from the Console. (Access for APIs and Python Libraries is in a separate release)
The new "Projects" component replaces "Tenant" as the main organizational concept in Cortex.
Administrators may grant users access to Projects from the CLI.
Users can save and open projects they have access to from the Console and CLI.
Administrators have access to the FabricOps utility to deploy Cortex actions as Docker images and pull down Cortex resources (agent, skills, actions) driven by a manifest file. This can be used by your CI/CD pipeline to deploy Cortex resources in a new environment such as testing, staging.

note

To install the CLI for this Preview release run:

npm install cortex-cli@2.0.0-a.1

Remember this is a Preview release, so you will likely encounter problems trying to do all things you are used to doing in SENSA Fabric. In particular Fabric Console, Fabric Docs, APIs, and Python Libraries are under construction. Please be patient, GREAT things are on the way.

note

In this release Developers and Administrators are able to create, list, and get all projects. However, developers who have not been granted access to a project won’t be able to see the resources in the project using the CLI.

The ability to create, list, view details, and manage resources for Projects from the Console will be updated next release to respect the new authorization settings.

October 13, 2020

DCI app version: 5.0.30, Cortex Helm Chart version 0.5.6, cortex-python library v1.4.1, cortex-python-profiles library v1.2.1, cortex-python-builders library v1.2.1

note

This version (5.0.30) is the final version of SENSA Fabric that is compatible with the standalone installation of Studio. When you upgrade to this release, we encourage you to begin using the new web-based Console that integrates Cortex components. It is available at https://console.<dci-base-domain>.

Improvements

Web-based Cortex: UI enhancements for Profiles and Studio
Documentation: Cortex python libraries have been improved and they can be accessed from the Reference Guides section of the docs. You can view them at the locations below:

Features

Documentation: OpenAPI specification
Web-based SENSA Fabric Console with an integrated tool suite (official release)

July 31, 2020

DCI app version: 5.0.29, Cortex helm chart version: 0.5.5, CLI: 1.1.1

Features

New public Fabric examples repo that provides use case example code. Follow the instructions in the README to get started.
New versioned Cortex Docs.
Admin Console Metrics dashboard has been replaced by a System Health Dashboard that can be used for troubleshooting cluster issues.

note

Web-based Studio and Profiles: Fabric Console now provides a single access point to SENSA Fabric tools including: Admin Portal, Studio (BETA), Profiles (BETA), and Fabric Docs.

Give it a try! Let your Cognitive Scale representative know if you do, and let us know how we can make it even better.

Deprecations

Cortex Studio (desktop): Cortex Studio desktop will be deprecated in the future in favor of the SENSA Fabric web-based Console that includes Studio tools. Your Studio local application will continue to work, but try out the new web-based portal; we think you'll like it!
CLI: The cortex skills generate command is now deprecated. For help getting started when building new skills, use use the sample code provided in the new cortex-fabric-examples repo.

July 2, 2020

cortex-python-profiles v1.2.0, and cortex-python-builders v1.2.0

Both the cortex-python-profiles and cortex-python-builders libraries have been updated as follows:

The client instance created from cortex.client() can now be passed into any of the client classes provided with the Cortex python library.
Added support for configuring a Cortex client connection with custom SSL certificates.

June 30, 2020

DCI v5.0.28, Cortex helm chart v0.5.4

In this release, all Cortex services have been updated to address CVE vulnerabilities discovered during a routine security scan. If you are updating an existing cluster to this release, please see the cortex-charts documentation for upgrade notes.
Resolved a memory leak related to authentication, which improved performance across the system.

June 15, 2020

cortex-python v1.4.0

This week's release provides an update to the cortex-python library to improve how authentication works:

The client instance created from cortex.client() can now be passed into any of the client classes provided with the Cortex python library.
Added support for configuring a Cortex client connection with custom SSL certificates.

June 9, 2020

DCI v5.0.27, Cortex helm chart v0.5.3, Studio v5.2.624

The new version of Cortex Studio removes the beta flag for Profile-of-One.
The node version prerequisite has been changed to >10.
Cortex services have been updated to cortex-express-common 1.5.0 to fix vulnerabilities.
Connections service has been updated to ensure that secrets are not returned in the get connection response.
For daemon actions, non-200 responses are now returned as expected rather than being converted to 500s.
A new documentation site is being released for SENSA Fabric docs at cognitivescale.github.io/cortex-fabric. When you upgrade your DCI instance to this release, any links to docs.<base-url> for your cluster will be redirected to the public docs site by default.

May 1, 2020

SENSA Fabric operator v1.0.0

SENSA Fabric v1.0.0 operator released to RedHat OpenShift Marketplace. Customers can now [install SENSA Fabric](/cortex-fabric/getting-started/install-rhos-operator on an OpenShift 4.2 cluster seamlessly.

April 9, 2020

cortex-python v1.3.0

This week's release only includes an update to the cortex-python library:

The library now provides more details when an error occurs.
The version range of the PyYAML dependency in the cortex-python library has been updated to resolve a PyYAML vulnerability (CVE-2020-1747).

April 1, 2020

DCI v5.0.26, Cortex helm chart v0.5.1

New feature

Added support for providing custom SMTP configurations via the Cortex Helm Chart.

March 30, 2020

DCI v5.0.25, Cortex helm chart v0.5.0

New features

With the new podspec feature, you can now more directly manage action resources, such as GPU, memory, or CPU. The podspec feature works by patching the kubernetes PodTemplateSpec generated by Fabric.
Profile-of-one has been updated to support inserting profile attributes in bulk.
Support for non-root installation of the CLI.

Deprecations and migrations

The CLI action deploy options --memory and --vcpu are no longer available. Instead use the podspec option to customize pod resources for actions at runtime.
When you use the podspec option to configure an action's resources, action metrics are externalized from Fabric Console and are not viewable there. To have access to logs and metrics, you must use a third party aggregator like Grafana, Cloud Watch, or Azure Dashboards. Logging is delegated to a log aggregator defined on the cluster and will NOT be provided in Fabric Console. See the cortex-charts documentation for cluster configuration options.
Functions have been formally deprecated and removed from the documentation. Use daemons in place of functions.
Profile-of-one has migrated from a single event loading paradigm to a bulk loader, which is able to process multiple profile events in a single job. The following is a summary of the changes that resulted from this shift in paradigm:
- The underlying data model changed to easily allow the bulk insertions of profile attributes against the underlying Po1 database.
- The way profiles are versioned changed, allowing multiple attributes to be updated per version.
- The association between Profile Attribute and Specific Profile Schemas was removed, so the profiles no longer need to rebuild every time a Profile Schema changes.

February 27, 2020

DCI v5.0.24, Cortex helm chart v0.4.1

For RedHat OpenShift 3.11 deployments, special privileges and/or run-as-root is not required to install and configure clusters under the OpenShift security context, non-root.
Cortex alpine-based images have been deprecated in favor of more secure UBI8-based images for all deployments.

February 17, 2020

DCI v5.0.23, Cortex helm chart v0.4.0

The images provided for cortex-charts installations have been updated as follows:

UBI images updated from UBI 7 to UBI 8 for improved security.
Several images have been updated to ensure that Redis sentinels are optimally configured.
The https://api.<dci-base-domain>/v2/info endpoint has been updated to ensure that it accurately reports on the health of the system.

April 24, 2023 - v6.4.0​

Console​

Profiles SDK​

Runtime​

Campaigns​

CLI​

Cortex-Python​

Fabric-ops​

Accounts / RBAC​

Housekeeping​

Examples​

Upgrades​

Resolved Issues​

Known Issues​

October 21, 2022 - v6.3.3​

Profiles SDK​

Fabric Console​

Resolved Issues​

Known Issues​

August 8, 2022 - v6.3.0​

Fabric CLI​

Fabric Console​

Cortex Profile SDK​

Resolved Issues​

Known Issues​

June 21, 2022 - v6.2.2​

Resolved Issues​

Known Issues​

May 21, 2022 - v6.2.1​

Resolved Issues 6.2.1​

Known Issues 6.2.1​

May 6, 2022 - v6.2.0​

Resolved Issues 6.2.0​

Known Issues 6.2.0​

March 30, 2022 - v6.1.1​

Backend​

CLI​

Console​

SENSA Fabric Examples​

Known Issues Resolved​

Known Issues​

November 29, 2021 - v6.0.10​

Documentation​

Known Issues Resolved​

Known Issues (in this release)​

Oct 13, 2021 Release version 6.0.9​

Backend​

CLI​

Fabric Console​

SENSA Fabric Examples​

Python Library​

Known Issues​

August 19, 2021 Release v6.0.8​

Backend​

CLI Commands​

Console​

SENSA Fabric Examples (GitHub repo)​

Python Library​

Known Issues​

June 30, 2021 Release v6.0.7​

Backend​

Console​

May 11, 2021 Release v6.0.5​

Backend​

Console​

Documentation​

Bug Fixes​

April 7, 2021 Release v6.0.3​

Backend​

Console​

Documentation​

Bug Fixes​

March 4, 2021 - Release v6.0.2​

Backend​

Console​

Documentation​

Bug fixes​

Known issues​

December 16, 2020 - Release v6.0.1​

Features​

April 24, 2023 - v6.4.0

Console

Profiles SDK

Runtime

Campaigns

CLI

Cortex-Python

Fabric-ops

Accounts / RBAC

Housekeeping

Examples

Upgrades

Resolved Issues

Known Issues

October 21, 2022 - v6.3.3

Profiles SDK

Fabric Console

Resolved Issues

Known Issues

August 8, 2022 - v6.3.0

Fabric CLI

Fabric Console

Cortex Profile SDK

Resolved Issues

Known Issues

June 21, 2022 - v6.2.2

Resolved Issues

Known Issues

May 21, 2022 - v6.2.1

Resolved Issues 6.2.1

Known Issues 6.2.1

May 6, 2022 - v6.2.0

Resolved Issues 6.2.0

Known Issues 6.2.0

March 30, 2022 - v6.1.1

Backend

CLI

Console

SENSA Fabric Examples

Known Issues Resolved

Known Issues

November 29, 2021 - v6.0.10

Documentation

Known Issues Resolved

Known Issues (in this release)

Oct 13, 2021 Release version 6.0.9

Backend

CLI

Fabric Console

SENSA Fabric Examples

Python Library

Known Issues

August 19, 2021 Release v6.0.8

Backend

CLI Commands

Console

SENSA Fabric Examples (GitHub repo)

Python Library

Known Issues

June 30, 2021 Release v6.0.7

Backend

Console

May 11, 2021 Release v6.0.5

Backend

Console

Documentation

Bug Fixes

April 7, 2021 Release v6.0.3

Backend

Console

Documentation

Bug Fixes

March 4, 2021 - Release v6.0.2

Backend

Console

Documentation

Bug fixes

Known issues

December 16, 2020 - Release v6.0.1

Features