Skip to main content
Version: latest

Manage Data Sources

This is a guide to configuring Data Sources using the Fabric Console.

Configuring Data Sources is a prerequisite to building profile schemas and campaigns.

A data source ingests the data from the selected connection for the configured attributes.

Prerequisites​

  1. Define Connection Secrets
  2. Define Connections

Define Data Sources in Console​

  1. Log in to the Fabric Console at https://api.<dci-base-domain>; where <dci-base-domain> is the domain set up for your Dedicated Cortex Instance.

  2. Select from the Projects you have been granted access to.

  3. In the left navigation panel click Data Sources.

  4. Click Add Data Source button at the top right.

  5. Define the data source metadata:

    • Enter a Data Source Title
    • Review the auto-generated Name for the Data Source. By default this is auto-generated. You may toggle the auto-generated switch to off and edit the name or enter a unique URL-friendly name. Data Source names must be alphanumeric, beginning with a letter and ending with a letter or number. Underscores may be used, but no spaces or other special characters are allowed. (Names must be 20 characters or fewer unless otherwise configured in the Helm chart.)
    • Select a Connection that you have predefined.

  6. Click Next at the bottom left to open the Attributes selection page.

  7. On the left side of the page select the attributes from the connection that you want to include in the Data Source.

    On the right side of the page you can view a preview of the data source and stats relating to each of the attributes.

    caution

    Attributes/Column names must be alphanumeric, beginning with a letter and ending with a letter or number. Underscores may be used, but no spaces or other special characters are allowed.

  1. Select the primary key for the data source (required). Primary Key must be an attribute that has unique values.
  2. Click Save at the bottom left.

The Preview tab opens with the attributes you selected displayed as columns with several rows of data populated from the connection.

The new data source is added to the Data Sources list page as a "card". The card shows the progress of the data source ingestion via a status bar. The status bar will indicate if the ingestion was active and whether or not it completed successfully.

If the ingestion job fails, the tail of the job log is displayed in a toast at the top of the page to provide insight into the root-cause of the failure.

When the ingestion is complete the status "Completed" is displayed at the top right of the data source tile on the list page.

Data Source status​

The status of a Data source ingestion job is displayed in the Console on the Data Source list page at the top right of each card.

Statuses include:

  • SUBMITTED: The data ingest job has been submitted.
  • QUEUED: The ingest job has been queued.
  • ACTIVE: The job is active.
  • COMPLETE: The ingest job has completed.
  • FAILED/ERROR: The ingest job did not complete because of an error.

Refresh Data Sources​

To refresh or delete a data source open the vertical ellipsis menu on the top of the Preview tab or on a data source tile on the data source list page.

Changes to the backend data from the connection are not reflected in the data source until the data source is refreshed. Select Refresh to trigger a re-ingestion process for updating data for the data source.

To refresh a Data Source:

  1. Authenticate to Console and select your Project context.

  2. Click Data Sources in the left navigation panel.

  3. Find the Data Source you wish to refresh, open the kebab menu on the top right end of the card, and select Refresh Data Source.

    You may also click the row to open the Data Source details page. At the top right open the kebab menu and select Refresh Data Source.

  4. When you select Refresh, a validation is run, and the status is changed to SUBMITTED.

    If the validation shows an error state, the error icon is displayed. Click the icon to view details for troubleshooting.

  5. The refresh job is initiated and the status changes to ACTIVE.

  6. When the refresh job has succeeded, the status changes to COMPLETE

caution

Profiles must be rebuilt manually to ingest new data from Data Sources. Rebuild your downstream Profiles to ingest refreshed data in the Data Source.

Delete Data Sources​

To Delete a Data Source:

  1. Authenticate to Console and select your Project context.

  2. Click Data Sources in the left navigation panel.

  3. Find the Data Source you wish to refresh, open the kebab menu on the top right end of the card, and select Delete Data Source.

    You may also click the row to open the Data Source details page. At the top right open the kebab menu and select Delete Data Source.

  4. When you select Delete, an Impact Analysis is run to find the downstream impacts. If impacts are discovered, the deletion is NOT allowed. An error message is displayed identifying the impacts.

Changelog​

The Changelog tab provides a list of data source refreshes and edits. Each changelog entry provides:

  • The userid of the system user who made the change.
  • A timestamp of when the change was made.
  • The data source version (incremented each time the data source is edited or refreshed)
  • The operation that occurred. (WRITE, UPDATE, or DELETE)

Best Practices for creating Data Sources for Mission Runtime​

To display KPI statistics prior to runtime you may want to create a static Data Source that corresponds to a Connection (file) that contains previous feedback data and attributes.

For Example:

As a best practice, an S3 file Connection named member_feedback was configured.Configure a Data Source (Member Feedback Data Source) that references that Connection. Be sure to select the feedback attributes that are prepended with the mission_name.

Next, configure a Profile that references the Member Feedback Data Source to generate the Profile Schema.

note

A system-generated Connection and corresponding Data Source is added to your Project at Mission runtime to manage feedback and rewards. Runtime KPI data is collected in these components and displayed in the Campaign Dashboard regardless of weather or not you configure the static Connection and Data Source described above. You cannot/should not edit this Data Source.