Monitor Missions and Campaigns

This is a guide to monitoring missions and campaigns.

Monitor Runtime progress

Runtime may be monitored in the following ways:

• View the KPI measure of the Mission goals in the Campaign Dashboard display. As the Mission runs the score should trend upward since the model is continuously trained to ensure optimal results. (See details below)

  NOTE

  If you have not configured a static Connection/Data Source/Profile with the feedback attributes, the Campaign Dashboard will NOT display an initial KPI measure. Instead, an error will be displayed until the first Cohort subset completes the runtime cycle, creating the system-generated feedback Connection/Data Source/Profile and returning the data that can be used to calculate the KPI.

• View the system-generated Feedback Profile list-view that shows a unique count of the Profiles that have been cycled through. (Be sure to view the feedback profile that is created by the system durning runtime.)

  

• Query the activationID (described below) to view the runtime status. The status remains as "PENDING" until all Cohort Profiles have been cycled through.

• Run the "Monitor Mission Execution" (Python Lib) scripts provided below.

• Download the Mission log file from Managed Content.

View the Campaign Dashboard

After you have run a simulation or deployed a campaign, you can monitor its progress toward the KPI in the Campaign Dashboard in Console.

NOTE

Only the latest simulation or runtime data is reflected on the Dashboard.

1. Log in to Fabric Console.
2. Verify that the Project context is correct, and select a new Project if necessary. Click the Project button at the top of the left navigation panel and in the Projects list modal click the name of the Project under which the Campaign you wish to monitor is scoped.
3. Click Campaigns in the left navigation panel.
4. A list of Campaign tiles is displayed. Click the Campaign tile you wish to monitor.
5. The Campaign summary (Cohorts, Goals, and Missions) tab opens by default. Above the Campaign component tiles, the SUMMARY and DASHBOARD tabs are displayed. Click DASHBOARD.
6. The Dashboard display includes the following for active missions:

• The Campaign Name and Description
• The Mission status (in the orange pill):
  • In Progress (KPI details reflect simulation results)
  • Deployed (KPI details reflect deployment results)
• The Campaign Goal
• KPI details for the last simulation or deployment that include:
  • KPI status (in the green pill)
    • On Track
    • At Risk
  • The KPI percent achieved (in large font)
  • The (daily, weekly, monthly, or yearly) target (specified in the KPI configuration)
  • The (daily, weekly, monthly, or yearly) change from the last simulation or deployment run: + or - a percent of change
  • The Cohort associated with the KPI (specified in the KPI configuration)

Query Activation

As with any Cortex Fabric Agent, when the actions are complete the ActivationId and success boolean are returned.

{

"success": true,

"activationId": "d873f7e7-6646-4e45-bb9b-xxxx"

}

To query the details of the activation run:

cortex agents get-activation <activationId>

Response:

{

"success": true,

"requestId": "d873f7e7-6646-4e45-bb9b-xxxx",

"agentName": "cortex/sucess-job-agent",

"serviceName": "router_service",

"sessionId": "f2e8d101-dc4d-4017-a00f-xxxx",

"projectId": "jl01262021",

"username": "jlara@example.com",

"payload": {

"payload": {

"campaign_name": "myCampaign",

"mission_name": "myMission",

"profile_schema": "schemaName",

"watchlist_location":"s3://bucket-uri",

"batch_size": 10

}

},

"start": 1612889465658,

"status": "COMPLETED",

Monitor Mission Execution Scripts

1. Run the following python query:

   import requests

   import pandas as pd

   import cortex.utils as utils

   import urllib3

   urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

   # get current config using default profile

   fabric_config = utils.get_cortex_profile()

   # generate a JWT

   token = utils.generate_token(fabric_config)

   project = 'myProject'

   requestId = 'xxxxx'

   baseurl = f'{fabric_config.get("url")}/fabric/v4/projects/{project}/activations'

   headers = {

   'content-type': 'application/json',

   'Authorization': f'bearer {token}'

   }

   def getActivation(id):

   url = f'{baseurl}/{id}'

   r = requests.get(url, headers = headers, params={'report': True}, verify=False, allow_redirects=False);

   return r.json()

   def getInvoked(id):

   query = {

   'correlationId': id,

   'limit': 10000

   }

   r = requests.get(baseurl, headers = headers, params = query, verify=False, allow_redirects=False);

   return r.json()

   def printRun(id, tab):

   activation = getActivation(id)

   if not activation["success"]:

   print(activation["message"])

   exit(1)

   print(f'{tab}{activation["status"]} {id}')

   for t in activation["transits"]:

   print(f'{tab}- {t["title"]} {t["name"]} {t["status"]}')

   next = getInvoked(id)

   for i in next["activations"]:

   printRun(i["activationId"], tab+" ")

   printRun(requestId, "")

   Where:

   • project is your Project name.
   • requesId is the activationId you want query.
   • limit is the number of records you want to view.

Example Response:

Click to view the Example Response

PENDING cc8076e6-c581-4b3c-b4e0-445499a73728

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING f30722a5-89c3-4b14-99e1-283c9e2fbeba

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 7eab119f-e097-4c7b-9799-c95b5ec98f5b

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING d4562b93-3852-41c0-9b44-b58aa7d6fc1a

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 6d8578cc-fe66-40ef-881b-388d99548e56

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 179ef519-b6e4-44a0-a468-b1ec1dde9003

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 9e30221e-a5b0-4c2d-b90d-89a2115e4a79

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 4100b297-2db8-4302-ba81-98d0e9664fbd

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 216de92f-d497-45e4-9faa-be4a2882a052

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING d05df355-72ef-4e3a-b679-68913ca07277

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 10522062-2069-4ee5-9b91-1db481033303

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING d19efecb-5d1f-4b03-b7b5-5f1177e0af75

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 75c39cc7-601c-4099-bcb5-6638c9bfe266

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 013722c6-e82f-4019-bbf6-4795160a64ca

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 2795a227-5bb8-41a7-9b01-096c2a66db34

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING da735025-7493-4ec1-83a6-453c3cd0ecf9

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 6cdeb93b-d0eb-4962-8664-3c2fa41144c6

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING ea444612-e39f-4ce2-b6e8-07eb43d1b581

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING e9106a0b-688c-41a3-8300-5b47ede7669e

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 375c5f6c-903b-448d-bfcb-6320740a51a3

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 9211a9b1-e98f-4c29-85a9-c3beaab7e7b7

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 743a317a-698c-4d89-924c-d75ff5643155

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 2c1c81cc-1efa-4a84-9ef1-385d4a3ad161

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING bee09616-f1a2-4526-a59a-fc5329ee8542

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING a099fff7-ccfb-4e07-b410-b6bf19b2ab23

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 0451f1dc-de40-4a26-8d1d-cd07e1bcb173

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 28fa8e07-fa60-4191-b504-3ba93e3acf89

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING ab9332b2-975f-46ff-b41a-f8e0eaac5f95

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 9ceb0b66-8e2d-4207-8225-ae0144319367

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING d05f7658-d9df-4c52-8704-d593cd8bbfeb

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 9cde0e96-084c-45bf-bf6e-7d4636baa656

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 3c370bf3-7a49-49ad-813e-104a52453d6b

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 7e04cde9-29ee-40e3-b97c-c460a8d861fb

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING e7ce9335-2e35-40bb-b020-8b8e04d993c2

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 6b3647f9-f337-41b0-a5da-76570dd16671

- end_output end_service COMPLETE

- hrfsm-cm hrfsm-cm COMPLETE

- hrfsm-sr hrfsm-sr COMPLETE

- Agent invoke agent-invoke STARTED

- feedback_skill_input skill_router_service COMPLETE

PENDING 0917a3e5-3989-47b8-a515-d053098d5aab

- hrfsm-fs hrfsm-fs COMPLETE

- hrfsm-rs hrfsm-rs COMPLETE

- rewards_skill_output rewards_service COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 3605d753-cd4f-4242-9c63-372c118f653c

- hrfsm-ol hrfsm-ol COMPLETE

- Agent invoke agent-invoke STARTED

PENDING 38afd6c5-f624-4d53-adbe-cc8a7f7746df

- hrfsm-cm hrfsm-cm COMPLETE

2. Next, grep the processor-gateway logs by running the following command:

stern --context arn:aws:eks:us-east-1:<id>:cluster/<cluster-name> -n cortex cortex-processor-gateway -c cortex-processor-gateway | grep <requestId> \

+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway \

+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway \

+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway

NOTE: When you have some sort of controller in k8s, (e.g. Deployment/Job/Replicaset/CronJob) the resource used by the controller contains a template, which is used to spawn pods. The hash is the unique identifier that the k8s controller creates to track the pods internally. For example deployment.apps/cortex-processor-gateway creates replicaset replicaset.apps/cortex-processor-gateway-765c78d9fb which then creates pods.

In k8s you can find the hashes displayed like this:

namespace	pod	# of pods	status	# commpleted	elapsed time
cortex	pod/cortex-processor-gateway-765c78d9fb-cdpcv	2/2	Running	0	18h
cortex	pod/cortex-processor-gateway-765c78d9fb-p5fnh	2/2	Running	0	18h
cortex	pod/cortex-processor-gateway-765c78d9fb-zkps9	2/2	Running	0	6d15h

Example Response

Click to view the Example Response

cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"skill.output event for hrfsm ","@timestamp":1632878732178,"@fields":{"eventType":"skill.output","agentName":"hrfsm","channelId":"05f40760-4437-412a-8fe2-469338a78ff9","outputName":"end_output","projectId":"mctest19","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","serviceName":"cohort_manager_service","messageId":"05f40760-4437-412a-8fe2-469338a78ff9:1","username":"swinchester@cognitivescale.com","level":"info","label":"agent-events"}}

cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"No mapping found for \"05f40760-4437-412a-8fe2-469338a78ff9\" and output \"end_output\" in agent \"hrfsm\"","@timestamp":"2021-09-29T01:25:32.181Z","@fields":{"activationId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","agentName":"hrfsm","channelId":"05f40760-4437-412a-8fe2-469338a78ff9","projectId":"mctest19","sessionId":"ce5de740-bbc8-4834-b14c-695c4ff0cd9a","agent":"hrfsm","level":"warn","label":"gateway"}}

cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"agent.output event for hrfsm ","@timestamp":1632878732178,"@fields":{"eventType":"agent.output","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","projectId":"mctest19","sessionId":"ce5de740-bbc8-4834-b14c-695c4ff0cd9a","agentName":"hrfsm","error":"No mapping found for \"05f40760-4437-412a-8fe2-469338a78ff9\" and output \"end_output\" in agent \"hrfsm\"","level":"error","label":"agent-events"}}

cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"Task callback failed 38afd6c5-f624-4d53-adbe-cc8a7f7746df: Unexpected end of JSON input","@timestamp":"2021-09-29T01:25:38.137Z","@fields":{"activationId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","username":"swinchestero@cognitivescale.com","level":"error","label":"gateway"}}

cortex-processor-gateway-765c78d9fb-8lbmv cortex-processor-gateway {"@message":"agent.input event for hrfsm ","@timestamp":"2021-09-29T01:25:27.764Z","@fields":{"eventType":"agent.input","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","projectId":"mctest19","sessionId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","agentName":"hrfsm","level":"info","label":"agent-events"}}

cortex-processor-gateway-765c78d9fb-8lbmv cortex-processor-gateway {"@message":"skill.input event for mctest19-hrfsm hrfsm-cm","@timestamp":1632878727781,"@fields":{"eventType":"skill.input","agentName":"hrfsm","channelId":"92733052-7085-401b-a035-c589cfd0bbef","serviceName":"cohort_manager_service","projectId":"mctest19","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","sessionId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","correlationId":"3605d753-cd4f-4242-9c63-372c118f653c","level":"info","label":"agent-events"}}