Skip to main content
Version: 6.4.0

Skill and Agent Logs

This is a guide to searching and viewing Agent and Skill runtime logs from the processor-gateway service using kubectl.

Runtime orchestration for SENSA Fabric happens in Kubernetes. System administrators can configure 3rd party logging tools like Splunk, Kibana, or Elasticsearch following the instructions in the Cortex Charts documentation.

If a log aggregator has not been configured, users can view and query unformatted runtime log data coming from the processor-gateway service using kubectl.

Prerequisites​

  • Kubectl is installed
  • Agents and/or Skills have been configured and invoked successfully in your SENSA Fabric instance.

Query processor-gateway runtime logs​

To get all the stored logs for Agent runtime activations in the processor-gateway service run the following kubectl command:

kubectl logs deployment/cortex-processor-gateway -n cortex

You can filter your processor-gateway log query by any of the Agent fields provided in the response. The example below filters by Project ID and Agent name.

Query Example

kubectl logs deploy/cortex-processor-gateway -n cortex | grep '"agentName":"cortex/hello_agent"'

Response

{"@message":"agent.input event for cortex/hello_agent ","@timestamp":1624540380083,"@fields":{"eventType":"agent.input","requestId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","agentName":"cortex/hello_agent","serviceName":"input","channelId":"0b4577a2-3b51-4e53-9ee9-89a13bc0ef34","sessionId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","projectId":"johan","username":"cortex@example.com","properties":{},"payload":{"text":""},"level":"info","label":"agent-events"}}
{"@message":"skill.input event for johan-cortex-hello-agent hellodaemon","@timestamp":1624540380083,"@fields":{"agentName":"cortex/hello_agent","channelId":"0b4577a2-3b51-4e53-9ee9-89a13bc0ef34","serviceName":"input","payload":{"text":""},"projectId":"johan","requestId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","sessionId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","properties":{},"level":"info","label":"agent-events"}}
{"@message":"skill.output event for johan-cortex-hello-agent hellodaemon","@timestamp":1624540380266,"@fields":{"agentName":"cortex/hello_agent","channelId":"6697e03c-cca0-4969-bd41-86ce00968455","serviceName":"input","payload":{"message":"expect payload with text "},"projectId":"johan","requestId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","sessionId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","properties":{},"level":"info","label":"agent-events"}}
{"@message":"agent.output event for johan-cortex-hello-agent ","@timestamp":1624540380266,"@fields":{"requestId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","projectId":"johan","sessionId":"cdc9bb49-3bac-4378-8b44-a3def6e3a9cc","agentName":"cortex/hello_agent","payload":{"message":"expect payload with text "},"level":"info","label":"agent-events"}}
{"@message":"agent.input event for cortex/hello_agent ","@timestamp":1624540384760,"@fields":{"eventType":"agent.input","requestId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","agentName":"cortex/hello_agent","serviceName":"input","channelId":"0b4577a2-3b51-4e53-9ee9-89a13bc0ef34","sessionId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","projectId":"johan","username":"cortex@example.com","properties":{},"payload":{"text":""},"level":"info","label":"agent-events"}}
{"@message":"skill.input event for johan-cortex-hello-agent hellodaemon","@timestamp":1624540384760,"@fields":{"agentName":"cortex/hello_agent","channelId":"0b4577a2-3b51-4e53-9ee9-89a13bc0ef34","serviceName":"input","payload":{"text":""},"projectId":"johan","requestId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","sessionId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","properties":{},"level":"info","label":"agent-events"}}
{"@message":"skill.output event for johan-cortex-hello-agent hellodaemon","@timestamp":1624540384954,"@fields":{"agentName":"cortex/hello_agent","channelId":"6697e03c-cca0-4969-bd41-86ce00968455","serviceName":"input","payload":{"message":"expect payload with text "},"projectId":"johan","requestId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","sessionId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","properties":{},"level":"info","label":"agent-events"}}
{"@message":"agent.output event for johan-cortex-hello-agent ","@timestamp":1624540384954,"@fields":{"requestId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","projectId":"johan","sessionId":"4de4fa93-8346-41a4-843a-6ee7a1908bb8","agentName":"cortex/hello_agent","payload":{"message":"expect payload with text "},"level":"info","label":"agent-events"}}

Log details​

When you run the kubectl logs command, the following details are returned. (These are the same details returned when you run an Agent in Fabric Console.):

  • "activationId" - same as requestId identifier for the output of an Agent
  • "agentName" - the unique system name for the Agent or Skill
  • "serviceName" - the name of the input service
  • "channelId" - the Skill mapping identifier
  • "requestId" - the identifier for the processor-gateway run
  • "sessionId" - same as the Request ID and Activation ID unless specified in the input payload to get cached session data
  • "projectId" - the name of the project the Agent belongs to
  • "username" - email of the user who invoked the Agent
  • "level" - severity level, either "info" or "warn" ("warn" indicates that an error occurred)
  • "label" - the type of event ("agent-events")
  • "outputName" - name of the output service
  • "inputName "- name of the input service
  • "error" - error message if one is applicable
  • "payload" - array with the agent/skill input payload
  • "properties" - array with the configured agent/skill properties
  • "status" - Activation (and Transit) status: PENDING (waiting to run), STARTED (queued to begin), COMPLETE (running or finished), ERROR (not running/not complete)

Filter Query by requestId (activationId)​

To query runtime logs for an Agent you may want to filter by requestId (or activationId; they are the same). You can get this identifier by querying the CLI.

  1. Authenticate to the CLI.
  2. Request a list of Agents from your Project.
    cortex agents list --project myProject
  3. Use the name of an Agent from the list to request activationIds (the same as requestIds) for the agent.
    cortex agents list-activations agentName --project myProject

Log query with filter​

Example

 kubectl logs deployment/cortex-processor-gateway -n cortex | grep 9b5e345e-19f1-4686-8883-a6d9d2e5e67b

Response

{"@message":"agent.input event for spark-bulk-bat-21bcf ","@timestamp":1623412552354,"@fields":{"eventType":"agent.input","requestId":"9b5e345e-19f1-4686-8883-a6d9d2e5e67b","agentName":"spark-bulk-bat-21bcf","serviceName":"claims_classifier","channelId":"28e12c12-c966-42c3-a8fb-54f19d886e46","sessionId":"9b5e345e-19f1-4686-8883-a6d9d2e5e67b","projectId":"spark-pred","username":"cortex@example.com","level":"info","label":"agent-events"}}
{"@message":"skill.input event for spark-pred-spark-bulk-bat-21bcf claims-classif-a3325","@timestamp":1623412552354,"@fields":{"agentName":"spark-bulk-bat-21bcf","channelId":"28e12c12-c966-42c3-a8fb-54f19d886e46","serviceName":"claims_classifier","projectId":"spark-pred","requestId":"9b5e345e-19f1-4686-8883-a6d9d2e5e67b","sessionId":"9b5e345e-19f1-4686-8883-a6d9d2e5e67b","level":"info","label":"agent-events"}}
{"@message":"skill.output event for spark-bulk-bat-21bcf spark-pred-claims-classif-a3325","@timestamp":1623412619853,"@fields":{"agentName":"spark-bulk-bat-21bcf","channelId":"c71fd312-ed17-4b01-b619-08060cc77fa2","outputName":"response","projectId":"spark-pred","requestId":"9b5e345e-19f1-4686-8883-a6d9d2e5e67b","serviceName":"claims_classifier","username":"cortex@example.com","level":"info","label":"agent-events"}}
{"@message":"agent.output event for spark-pred-spark-bulk-bat-21bcf ","@timestamp":1623412619853,"@fields":{"requestId":"9b5e345e-19f1-4686-8883-a6d9d2e5e67b","projectId":"spark-pred","sessionId":"f24e0e07-2ae8-4179-8e1c-955705d7e52d","agentName":"spark-bulk-bat-21bcf","level":"info","label":"agent-events"}}

Interpret Agent status​

The "level" for an Agent is either "info" or "warn". If warn is displayed then a runtime error occurred and an error message may or may not be provided.

You know when an Agent is completed because an "agent.output event" is displayed, as in the example above. When the Agent is in progress, the logs will show one or more "skill.output events" but not an "agent.output event".

List and Describe Skills Statuses​

Set deployStatus and actionStatus to true when you invoke the skill or agent using the --json parameter to display the skill status when you query the response (with cortex skills list or cortex skills describe skillName)

To view a list of skills and their statuses in the CLI run:

cortex skills list --project myProject

Example

cortex skills list --project johan

Response

┌──────────────┬───────────────┬─────────────────┬─────────────┬────────────────────┐──────────┐
│ Name         │ Title         │ Description     │ Modified    │ Author             │ Status   |  
├──────────────┼───────────────┼─────────────────┼─────────────┼────────────────────┤──────────┤
│ merge-test   │ merge-test    │ v6 job          │ a month ago │ cortex@example.com │ Deployed |
├──────────────┼───────────────┼─────────────────┼─────────────┼────────────────────┤──────────┤
│ hubblejob    │ hubble job    │ get hubble data │ 2 months ago│ cortex@example.com │ Deployed |
└──────────────┴───────────────┴─────────────────┴─────────────┴────────────────────┴──────────┘

To view the details (or description) of a Skill including its statuses in the CLI run:

cortex skills describe skillName --project myProject

Example

cortex skills describe hubble-job --project myProject

Skill Statuses​

Skill have the following Deploy statuses:

  • Deployed: Saved in Cortex AND running on Kubernetes
  • Not deployed: Save in Cortex BUT not running on Kubernetes

Daemon Statuses​

In addition, Skill with daemons can have the following Action statuses:

  • Ready: Indicates that the web-service is up and running, followed by the number of daemons running over the configured replica count (e.g. (1/1))
  • Not ready: Indicates that the service is not running in Kubernetes, followed by the number of daemons running over the configured replica count (e.g. (0/1))