Version: 6.3.3

Monitor Missions and Campaigns

This is a guide to monitoring missions and campaigns.

Monitor Runtime progress

Runtime may be monitored in the following ways:

  • View the KPI measure of the Mission goals in the Campaign Dashboard display. As the Mission runs the score should trend upward since the model is continuously trained to ensure optimal results. (See details below)

  • View the system-generated Feedback Profile list-view that shows a unique count of the Profiles that have been cycled through. (Be sure to view the feedback profile that is created by the system durning runtime.)

    Feedback Profile list

  • Query the activationID (described below) to view the runtime status. The status remains as "PENDING" until all Cohort Profiles have been cycled through.

  • Run the "Monitor Mission Execution" (Python Lib) scripts provided below.

  • Download the Mission log file from Managed Content.

View the Campaign Dashboard

After you have run a simulation or deployed a campaign, you can monitor its progress toward the KPI in the Campaign Dashboard in Console.

Campaign dashboard

  1. Log in to Fabric Console.
  2. Verify that the Project context is correct, and select a new Project if necessary. Click the Project button at the top of the left navigation panel and in the Projects list modal click the name of the Project under which the Campaign you wish to monitor is scoped.
  3. Click Campaigns in the left navigation panel.
  4. A list of Campaign tiles is displayed. Click the Campaign tile you wish to monitor.
  5. The Campaign summary (Cohorts, Goals, and Missions) tab opens by default. Above the Campaign component tiles, the SUMMARY and DASHBOARD tabs are displayed. Click DASHBOARD.
  6. The Dashboard display includes the following for active missions:
  • The Campaign Name and Description
  • The Mission status (in the orange pill):
    • In Progress (KPI details reflect simulation results)
    • Deployed (KPI details reflect deployment results)
  • The Campaign Goal
  • KPI details for the last simulation or deployment that include:
    • KPI status (in the green pill)
      • On Track
      • At Risk
    • The KPI percent achieved (in large font)
    • The (daily, weekly, monthly, or yearly) target (specified in the KPI configuration)
    • The (daily, weekly, monthly, or yearly) change from the last simulation or deployment run: + or - a percent of change
    • The Cohort associated with the KPI (specified in the KPI configuration)

Query Activation

As with any Cortex Fabric Agent, when the actions are complete the ActivationId and success boolean are returned.

{
"success": true,
"activationId": "d873f7e7-6646-4e45-bb9b-xxxx"
}

To query the details of the activation run:

cortex agents get-activation <activationId>

Response:

{
"success": true,
"requestId": "d873f7e7-6646-4e45-bb9b-xxxx",
"agentName": "cortex/sucess-job-agent",
"serviceName": "router_service",
"sessionId": "f2e8d101-dc4d-4017-a00f-xxxx",
"projectId": "jl01262021",
"username": "jlara@example.com",
"payload": {
"payload": {
"campaign_name": "myCampaign",
"mission_name": "myMission",
"profile_schema": "schemaName",
"watchlist_location":"s3://bucket-uri",
"batch_size": 10
}
},
"start": 1612889465658,
"status": "COMPLETED",

Monitor Mission Execution Scripts

  1. Run the following python query:

    import requests
    import pandas as pd
    import cortex.utils as utils
    import urllib3
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
    # get current config using default profile
    fabric_config = utils.get_cortex_profile()
    # generate a JWT
    token = utils.generate_token(fabric_config)
    project = 'myProject'
    requestId = 'xxxxx'
    baseurl = f'{fabric_config.get("url")}/fabric/v4/projects/{project}/activations'
    headers = {
    'content-type': 'application/json',
    'Authorization': f'bearer {token}'
    }
    def getActivation(id):
    url = f'{baseurl}/{id}'
    r = requests.get(url, headers = headers, params={'report': True}, verify=False, allow_redirects=False);
    return r.json()
    def getInvoked(id):
    query = {
    'correlationId': id,
    'limit': 10000
    }
    r = requests.get(baseurl, headers = headers, params = query, verify=False, allow_redirects=False);
    return r.json()
    def printRun(id, tab):
    activation = getActivation(id)
    if not activation["success"]:
    print(activation["message"])
    exit(1)
    print(f'{tab}{activation["status"]} {id}')
    for t in activation["transits"]:
    print(f'{tab}- {t["title"]} {t["name"]} {t["status"]}')
    next = getInvoked(id)
    for i in next["activations"]:
    printRun(i["activationId"], tab+" ")
    printRun(requestId, "")

    Where:

    • project is your Project name.
    • requesId is the activationId you want query.
    • limit is the number of records you want to view.

Example Response:

Click to view the Example Response
PENDING cc8076e6-c581-4b3c-b4e0-445499a73728
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING f30722a5-89c3-4b14-99e1-283c9e2fbeba
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 7eab119f-e097-4c7b-9799-c95b5ec98f5b
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING d4562b93-3852-41c0-9b44-b58aa7d6fc1a
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 6d8578cc-fe66-40ef-881b-388d99548e56
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 179ef519-b6e4-44a0-a468-b1ec1dde9003
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 9e30221e-a5b0-4c2d-b90d-89a2115e4a79
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 4100b297-2db8-4302-ba81-98d0e9664fbd
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 216de92f-d497-45e4-9faa-be4a2882a052
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING d05df355-72ef-4e3a-b679-68913ca07277
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 10522062-2069-4ee5-9b91-1db481033303
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING d19efecb-5d1f-4b03-b7b5-5f1177e0af75
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 75c39cc7-601c-4099-bcb5-6638c9bfe266
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 013722c6-e82f-4019-bbf6-4795160a64ca
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 2795a227-5bb8-41a7-9b01-096c2a66db34
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING da735025-7493-4ec1-83a6-453c3cd0ecf9
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 6cdeb93b-d0eb-4962-8664-3c2fa41144c6
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING ea444612-e39f-4ce2-b6e8-07eb43d1b581
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING e9106a0b-688c-41a3-8300-5b47ede7669e
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 375c5f6c-903b-448d-bfcb-6320740a51a3
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 9211a9b1-e98f-4c29-85a9-c3beaab7e7b7
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 743a317a-698c-4d89-924c-d75ff5643155
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 2c1c81cc-1efa-4a84-9ef1-385d4a3ad161
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING bee09616-f1a2-4526-a59a-fc5329ee8542
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING a099fff7-ccfb-4e07-b410-b6bf19b2ab23
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 0451f1dc-de40-4a26-8d1d-cd07e1bcb173
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 28fa8e07-fa60-4191-b504-3ba93e3acf89
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING ab9332b2-975f-46ff-b41a-f8e0eaac5f95
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 9ceb0b66-8e2d-4207-8225-ae0144319367
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING d05f7658-d9df-4c52-8704-d593cd8bbfeb
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 9cde0e96-084c-45bf-bf6e-7d4636baa656
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 3c370bf3-7a49-49ad-813e-104a52453d6b
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 7e04cde9-29ee-40e3-b97c-c460a8d861fb
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING e7ce9335-2e35-40bb-b020-8b8e04d993c2
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 6b3647f9-f337-41b0-a5da-76570dd16671
- end_output end_service COMPLETE
- hrfsm-cm hrfsm-cm COMPLETE
- hrfsm-sr hrfsm-sr COMPLETE
- Agent invoke agent-invoke STARTED
- feedback_skill_input skill_router_service COMPLETE
PENDING 0917a3e5-3989-47b8-a515-d053098d5aab
- hrfsm-fs hrfsm-fs COMPLETE
- hrfsm-rs hrfsm-rs COMPLETE
- rewards_skill_output rewards_service COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 3605d753-cd4f-4242-9c63-372c118f653c
- hrfsm-ol hrfsm-ol COMPLETE
- Agent invoke agent-invoke STARTED
PENDING 38afd6c5-f624-4d53-adbe-cc8a7f7746df
- hrfsm-cm hrfsm-cm COMPLETE
  1. Next, grep the processor-gateway logs by running the following command:
stern --context arn:aws:eks:us-east-1:<id>:cluster/<cluster-name> -n cortex cortex-processor-gateway -c cortex-processor-gateway | grep <requestId> \
+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway \
+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway \
+ cortex-processor-gateway-k8sControllerHash › cortex-processor-gateway

NOTE: When you have some sort of controller in k8s, (e.g. Deployment/Job/Replicaset/CronJob) the resource used by the controller contains a template, which is used to spawn pods. The hash is the unique identifier that the k8s controller creates to track the pods internally. For example deployment.apps/cortex-processor-gateway creates replicaset replicaset.apps/cortex-processor-gateway-765c78d9fb which then creates pods.

In k8s you can find the hashes displayed like this:

namespacepod# of podsstatus# commpletedelapsed time
cortexpod/cortex-processor-gateway-765c78d9fb-cdpcv2/2Running018h
cortexpod/cortex-processor-gateway-765c78d9fb-p5fnh2/2Running018h
cortexpod/cortex-processor-gateway-765c78d9fb-zkps92/2Running06d15h

Example Response

Click to view the Example Response
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"skill.output event for hrfsm ","@timestamp":1632878732178,"@fields":{"eventType":"skill.output","agentName":"hrfsm","channelId":"05f40760-4437-412a-8fe2-469338a78ff9","outputName":"end_output","projectId":"mctest19","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","serviceName":"cohort_manager_service","messageId":"05f40760-4437-412a-8fe2-469338a78ff9:1","username":"swinchester@cognitivescale.com","level":"info","label":"agent-events"}}
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"No mapping found for \"05f40760-4437-412a-8fe2-469338a78ff9\" and output \"end_output\" in agent \"hrfsm\"","@timestamp":"2021-09-29T01:25:32.181Z","@fields":{"activationId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","agentName":"hrfsm","channelId":"05f40760-4437-412a-8fe2-469338a78ff9","projectId":"mctest19","sessionId":"ce5de740-bbc8-4834-b14c-695c4ff0cd9a","agent":"hrfsm","level":"warn","label":"gateway"}}
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"agent.output event for hrfsm ","@timestamp":1632878732178,"@fields":{"eventType":"agent.output","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","projectId":"mctest19","sessionId":"ce5de740-bbc8-4834-b14c-695c4ff0cd9a","agentName":"hrfsm","error":"No mapping found for \"05f40760-4437-412a-8fe2-469338a78ff9\" and output \"end_output\" in agent \"hrfsm\"","level":"error","label":"agent-events"}}
cortex-processor-gateway-765c78d9fb-xr5cc cortex-processor-gateway {"@message":"Task callback failed 38afd6c5-f624-4d53-adbe-cc8a7f7746df: Unexpected end of JSON input","@timestamp":"2021-09-29T01:25:38.137Z","@fields":{"activationId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","username":"swinchestero@cognitivescale.com","level":"error","label":"gateway"}}
cortex-processor-gateway-765c78d9fb-8lbmv cortex-processor-gateway {"@message":"agent.input event for hrfsm ","@timestamp":"2021-09-29T01:25:27.764Z","@fields":{"eventType":"agent.input","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","projectId":"mctest19","sessionId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","agentName":"hrfsm","level":"info","label":"agent-events"}}
cortex-processor-gateway-765c78d9fb-8lbmv cortex-processor-gateway {"@message":"skill.input event for mctest19-hrfsm hrfsm-cm","@timestamp":1632878727781,"@fields":{"eventType":"skill.input","agentName":"hrfsm","channelId":"92733052-7085-401b-a035-c589cfd0bbef","serviceName":"cohort_manager_service","projectId":"mctest19","requestId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","sessionId":"38afd6c5-f624-4d53-adbe-cc8a7f7746df","correlationId":"3605d753-cd4f-4242-9c63-372c118f653c","level":"info","label":"agent-events"}}