Monitor Missions and Campaigns
This is a guide to monitoring missions and campaigns.
Monitor Runtime progress
Runtime may be monitored in the following ways:
View the KPI measure of the Mission goals in the Campaign Dashboard display. As the Mission runs the score should trend upward since the model is continuously trained to ensure optimal results. (See details below)
noteIf you have not configured a static Connection/Data Source/Profile with the feedback attributes, the Campaign Dashboard will NOT display an initial KPI measure. Instead, an error will be displayed until the first Cohort subset completes the runtime cycle, creating the system-generated feedback Connection/Data Source/Profile and returning the data that can be used to calculate the KPI.
View the system-generated Feedback Profile list-view that shows a unique count of the Profiles that have been cycled through. (Be sure to view the feedback profile that is created by the system durning runtime.)
Query the
activationID(described below) to view the runtime status. The status remains as "PENDING" until all Cohort Profiles have been cycled through.Run the "Monitor Mission Execution" (Python Lib) scripts provided below.
View the Campaign Dashboard
After you have run a simulation or deployed a campaign, you can monitor its progress toward the KPI in the Campaign Dashboard in Console.
note
Only the latest simulation or runtime data is reflected on the Dashboard.
- Log in to Fabric Console.
- Verify that the Project context is correct, and select a new Project if necessary. Click the Project button at the top of the left navigation panel and in the Projects list modal click the name of the Project under which the Campaign you wish to monitor is scoped.
- Click Campaigns in the left navigation panel.
- A list of Campaign tiles is displayed. Click the Campaign tile you wish to monitor.
- The Campaign summary (Cohorts, Goals, and Missions) tab opens by default. Above the Campaign component tiles, the SUMMARY and DASHBOARD tabs are displayed. Click DASHBOARD.
- The Dashboard display includes the following for active missions:
- The Campaign Name and Description
- The Mission status (in the orange pill):
- In Progress (KPI details reflect simulation results)
- Deployed (KPI details reflect deployment results)
- The Campaign Goal
- KPI details for the last simulation or deployment that include:
- KPI status (in the green pill)
- On Track
- At Risk
- The KPI percent achieved (in large font)
- The (daily, weekly, monthly, or yearly) target (specified in the KPI configuration)
- The (daily, weekly, monthly, or yearly) change from the last simulation or deployment run: + or - a percent of change
- The Cohort associated with the KPI (specified in the KPI configuration)
- KPI status (in the green pill)
Query Activation
As with any SENSA Fabric Agent, when the actions are complete the ActivationId and success boolean are returned.
{
"success": true,
"activationId": "d873f7e7-6646-4e45-bb9b-xxxx"
}
To query the details of the activation run:
cortex agents get-activation <activationId>
Response:
{
"success": true,
"requestId": "d873f7e7-6646-4e45-bb9b-xxxx",
"agentName": "cortex/sucess-job-agent",
"serviceName": "router_service",
"sessionId": "f2e8d101-dc4d-4017-a00f-xxxx",
"projectId": "jl01262021",
"username": "jlara@example.com",
"payload": {
"payload": {
"campaign_name": "myCampaign",
"mission_name": "myMission",
"profile_schema": "schemaName",
"watchlist_location":"s3://bucket-uri",
"batch_size": 10
}
},
"start": 1612889465658,
"status": "COMPLETED",
Monitor Mission Execution Scripts
Run the following python query:
import requests
import pandas as pd
import cortex.utils as utils
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# get current config using default profile
fabric_config = utils.get_cortex_profile()
# generate a JWT
token = utils.generate_token(fabric_config)
project = 'myProject'
requestId = 'xxxxx'
baseurl = f'{fabric_config.get("url")}/fabric/v4/projects/{project}/activations'
headers = {
'content-type': 'application/json',
'Authorization': f'bearer {token}'
}
def getActivation(id):
url = f'{baseurl}/{id}'
r = requests.get(url, headers = headers, params={'report': True}, verify=False, allow_redirects=False);
return r.json()
def getInvoked(id):
query = {
'correlationId': id,
'limit': 10000
}
r = requests.get(baseurl, headers = headers, params = query, verify=False, allow_redirects=False);
return r.json()
def printRun(id, tab):
activation = getActivation(id)
if not activation["success"]:
print(activation["message"])
exit(1)
print(f'{tab}{activation["status"]} {id}')
for t in activation["transits"]:
print(f'{tab}- {t["title"]} {t["name"]} {t["status"]}')
next = getInvoked(id)
for i in next["activations"]:
printRun(i["activationId"], tab+" ")
printRun(requestId, "")Where:
projectis your Project name.requesIdis the activationId you want query.limitis the number of records you want to view.
Example Response:
Click to view the Example Response
PENDING cc8076e6-c581-4b3c-b4e0-445499a73728
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING f30722a5-89c3-4b14-99e1-283c9e2fbeba
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 7eab119f-e097-4c7b-9799-c95b5ec98f5b
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING d4562b93-3852-41c0-9b44-b58aa7d6fc1a
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 6d8578cc-fe66-40ef-881b-388d99548e56
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 179ef519-b6e4-44a0-a468-b1ec1dde9003
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 9e30221e-a5b0-4c2d-b90d-89a2115e4a79
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 4100b297-2db8-4302-ba81-98d0e9664fbd
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 216de92f-d497-45e4-9faa-be4a2882a052
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING d05df355-72ef-4e3a-b679-68913ca07277
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 10522062-2069-4ee5-9b91-1db481033303
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING d19efecb-5d1f-4b03-b7b5-5f1177e0af75
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 75c39cc7-601c-4099-bcb5-6638c9bfe266
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 013722c6-e82f-4019-bbf6-4795160a64ca
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 2795a227-5bb8-41a7-9b01-096c2a66db34
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING da735025-7493-4ec1-83a6-453c3cd0ecf9
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 6cdeb93b-d0eb-4962-8664-3c2fa41144c6
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING ea444612-e39f-4ce2-b6e8-07eb43d1b581
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING e9106a0b-688c-41a3-8300-5b47ede7669e
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 375c5f6c-903b-448d-bfcb-6320740a51a3
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 9211a9b1-e98f-4c29-85a9-c3beaab7e7b7
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 743a317a-698c-4d89-924c-d75ff5643155
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 2c1c81cc-1efa-4a84-9ef1-385d4a3ad161
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING bee09616-f1a2-4526-a59a-fc5329ee8542
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING a099fff7-ccfb-4e07-b410-b6bf19b2ab23
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 0451f1dc-de40-4a26-8d1d-cd07e1bcb173
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 28fa8e07-fa60-4191-b504-3ba93e3acf89
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING ab9332b2-975f-46ff-b41a-f8e0eaac5f95
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 9ceb0b66-8e2d-4207-8225-ae0144319367
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING d05f7658-d9df-4c52-8704-d593cd8bbfeb
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 9cde0e96-084c-45bf-bf6e-7d4636baa656
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 3c370bf3-7a49-49ad-813e-104a52453d6b
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 7e04cde9-29ee-40e3-b97c-c460a8d861fb
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING e7ce9335-2e35-40bb-b020-8b8e04d993c2
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 6b3647f9-f337-41b0-a5da-76570dd16671
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 0917a3e5-3989-47b8-a515-d053098d5aab
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 3605d753-cd4f-4242-9c63-372c118f653c
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 38afd6c5-f624-4d53-adbe-cc8a7f7746df
- hrfsm-cm hrfsm-cm COMPLETE
- Next,
grepthe processor-gateway logs by running the following command:
stern --context arn:aws:eks:us-east-1:<id>:cluster/<cluster-name> -n cortex cortex-processor-gateway -c cortex-processor-gateway | grep <requestId> \
+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway \
+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway \
+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway
NOTE: When you have some sort of controller in k8s, (e.g. Deployment/Job/Replicaset/CronJob) the resource used by the controller contains a template, which is used to spawn pods. The hash is the unique identifier that the k8s controller creates to track the pods internally. For example deployment.apps/cortex-processor-gateway creates replicaset replicaset.apps/cortex-processor-gateway-765c78d9fb which then creates pods.
In k8s you can find the hashes displayed like this:
| namespace | pod | # of pods | status | # commpleted | elapsed time|
| --- | --- | --- | --- | --- | --- |
| cortex | pod/cortex-processor-gateway-765c78d9fb-cdpcv | 2/2 | Running | 0 | 18h |
| cortex | pod/cortex-processor-gateway-765c78d9fb-p5fnh | 2/2 | Running | 0 | 18h |
| cortex | pod/cortex-processor-gateway-765c78d9fb-zkps9 | 2/2 | Running | 0 | 6d15h |
Example Response
Click to view the Example Response
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"skill.output event for hrfsm ","@timestamp":1632878732178,"@fields":{"eventType":"skill.output","agentName":"hrfsm","channelId":"05f40760-4437-412a-8fe2-469338a78ff9","outputName":"end_output","projectId":"mctest19","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","serviceName":"cohort_manager_service","messageId":"05f40760-4437-412a-8fe2-469338a78ff9:1","username":"swinchester@cognitivescale.com","level":"info","label":"agent-events"}}
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"No mapping found for \"05f40760-4437-412a-8fe2-469338a78ff9\" and output \"end_output\" in agent \"hrfsm\"","@timestamp":"2021-09-29T01:25:32.181Z","@fields":{"activationId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","agentName":"hrfsm","channelId":"05f40760-4437-412a-8fe2-469338a78ff9","projectId":"mctest19","sessionId":"ce5de740-bbc8-4834-b14c-695c4ff0cd9a","agent":"hrfsm","level":"warn","label":"gateway"}}
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"agent.output event for hrfsm ","@timestamp":1632878732178,"@fields":{"eventType":"agent.output","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","projectId":"mctest19","sessionId":"ce5de740-bbc8-4834-b14c-695c4ff0cd9a","agentName":"hrfsm","error":"No mapping found for \"05f40760-4437-412a-8fe2-469338a78ff9\" and output \"end_output\" in agent \"hrfsm\"","level":"error","label":"agent-events"}}
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"Task callback failed 38afd6c5-f624-4d53-adbe-cc8a7f7746df: Unexpected end of JSON input","@timestamp":"2021-09-29T01:25:38.137Z","@fields":{"activationId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","username":"swinchestero@cognitivescale.com","level":"error","label":"gateway"}}
cortex-processor-gateway-765c78d9fb-8lbmv cortex-processor-gateway {"@message":"agent.input event for hrfsm ","@timestamp":"2021-09-29T01:25:27.764Z","@fields":{"eventType":"agent.input","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","projectId":"mctest19","sessionId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","agentName":"hrfsm","level":"info","label":"agent-events"}}
cortex-processor-gateway-765c78d9fb-8lbmv cortex-processor-gateway {"@message":"skill.input event for mctest19-hrfsm hrfsm-cm","@timestamp":1632878727781,"@fields":{"eventType":"skill.input","agentName":"hrfsm","channelId":"92733052-7085-401b-a035-c589cfd0bbef","serviceName":"cohort_manager_service","projectId":"mctest19","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","sessionId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","correlationId":"3605d753-cd4f-4242-9c63-372c118f653c","level":"info","label":"agent-events"}}