How to - Error handling

The goal is to present some best practices for error handling within Orchestrator.

We want to track the execution status of a workflow, intermediate or final.

We want to send an email notification on error or success, intermediate or final.

We also want to monitor via the Orchestrator API the execution status.

Note that Orchestrator is managing its own workflow execution status. Here we want to manage a custom defined status that can take any form such as YAML or XML or JSON or HTML, etc. The use case may be to send a custom email notification based on the status content at the workflow top level (rather than within each sub-workflow where an error can be produced).

The model workflow is the following:

 error_handling.JPG

2 actions are represented in this worflow, namely action 1 and action 2. They could be anything, like an individual transcode operation or a sub-workflow composed of multiple actions.

The custom execution status is stored in a shared location. In the proposed implementation, it is a text file, configured in the params step. On a Highly Available Orchestrator system, it must be on a shared storage. It could also be a record in a shared database table (e.g. shared_states).

The status file is different with each workorder to guarantee that multiple running instances of the same workflow (e.g. workflow with a trigger) don’t use the same shared location. The file name is built within the ‘status init’ step to include the workorder id as a suffix.

The status file is initialized here. The proposed status content format is composed of a status code and a status message, stored in yaml format.

statusfile = inputs['statusfile']
statusfile_new = File.join(File.dirname(statusfile),File.nakedname(statusfile)) + '_' + @work_order_id.to_s + File.extname(statusfile)
settings = {"status" => "00", "message" => "init"}
File.open(statusfile_new, 'w') {|f| f.write(settings.to_yaml) }
outputs['statusfile_new'] = statusfile_new

The file content is for example:

---
message: init
status: "00"

Each time the status must be kept (e.g. report error or success), the proposed code in a custom Ruby plugin is

settings = {"status" => "11", "message" => "action 1 error"}
File.open(inputs['statusfile'], 'w') {|f| f.write(settings.to_yaml) }

Or:

settings = {"status" => "00", "message" => "success"}
File.open(inputs['statusfile'], 'w') {|f| f.write(settings.to_yaml) }

The status reporting can be done anywhere like for example in a sub-workfow upon any event of choice, final or intermediate. It’s not necessarily done in the parent workflow as shown above.

To obtain the status (for example before sending an email notification), we read it from the shared file (e.g. in ‘read status and clean’ step) with this code in a custom Ruby plugin:

settings = YAML::load_file(inputs['statusfile'])
outputs['message'] = settings['message']
outputs['status'] = settings['status']

The status code and message are then in outputs and can be passed to the step needing them

The email notification may be a template with the same inputs and treatment for all needed notifications. It can also be a sub-workflow if more complex logic is required.

The status file must also be deleted at the end of the workflow (e.g. in the ‘read status and clean’ step) with:

File.delete(inputs['statusfile']) rescue nil

The status code could be obtained from Orchestrator API via the ‘fetch output of a workorder’ call such as:

http://<Orchestrator IP address>/aspera/orchestrator/workflow_reporter/work_order_output/<workorder id>.string?login=<login>&password=<password>&step_name=<step name>&variable_name=<output variable name>

This would return the status code such as:

success

Note that this output can only be obtained when the monitored step has been executed. Before, any output value will show up as blank. The API caller has to keep polling until the final step is reached. Instead of just the status code, the full status content could also be returned (in this case, just define one output variable and assign the whole status content to it).

If intermediate status reports are needed, a specific workflow may be built to fetch the content from the disk or database. The API call would then be a synchronous ‘initiate workorder’, returning the custom status upon completion.

Instead of status stored on file or database, it’s also possible to use the Orchestrator own status, customized via the 'Exit status' plugin. This status can be monitored via the 'fetch workorder status' API call. Note that the ‘Exit status’ can be applied only when a (sub)-workflow is finalized (not for intermediate steps).

Attachments

2 Comments

  • Avatar
    BO Engineering

    Hello Christophe,

    thank you for this workflow. Finally i had time to test it and i have an idea to improve it.

    instead of using 3 different CR steps for each status, i´d use one CR that has 3 inputs: statusfile, status, message

    code:

    settings = {"status" => inputs['status'], "message" => inputs['message']}

    File.open(inputs['statusfile'], 'w') {|f| f.write(settings.to_yaml) }

     

    BR

    maxxx

     

  • Avatar
    Christophe Thiebot

    Added some comments about the Exit status plugin use.

    And about how to get intermediate status reports via the Orchestrator API.

Please sign in to leave a comment.
Powered by Zendesk