Work Items

Information about the current status of running tasks and their rejections can be seen in the Work Items view under Administration > System Status > Work Items.  

Mango runs every task at one of three levels, High, Medium and Low.  Some tasks have priorities that can be changed, others do not.  Mango provides a way to monitor the various rejections of tasks in the System Status --> Work Items view, this document describes what this information means and how to tune Mango for the best performance.

Task Queues for the High and Medium pools are used to ensure that a given task runs in the order it was scheduled.  The ordering is based on a unique identifier and the queue can be limited or infinite.  Limited queues can have a hard limit defined in the code or rely on the env property runtime.realTimeTimer.defaultTaskQueueSize.  It should be noted that only the High and Medium pools can run Ordered tasks, however they can also run tasks that do not require an order.  If an un-ordered task is submitted to the pool it will run at some point in the future but not necessarily before other tasks of the same type.

Understanding Runtime Performance

Running Stats

This area provides insight into any tasks that are currently executing.  A task must be currently executing to be seen in this view, most tasks execute so quickly that this view will have little information in it in a healthy system.  When tasks start backing up and taking longer to run this page can provide insight into what tasks are long running and potentially degrading performance.

Rejected Stats

Information about task rejections can be found in this section. A task is rejected for one of 2 reasons.  First the queue may be full for an ordered task.  This means that the system cannot process the tasks fast enough and they have filled that task's specific queue and thus are being rejected.  This usually means that your timeouts are set improperly or the task is scheduled to execute too often.  Pool full rejections occur when the thread pool for that priority is full and no more task can be scheduled to run until a thread becomes free.

High Priority Tasks

High priority tasks run in the high priority pool at the highest thread priority of the Operating System and are ordered based on task.  For example tasks of different types may run at the same time while tasks such as the polling of a specific data source will run in its own queue such that the next poll can only happen after the current poll finishes.  Each high priority task has a maximum queue size (outlined below) that when full and a new task is attempted to be run it will be rejected with a Queue Full task rejection.

The High priority thread pool must be large enough to run all High priority tasks that are scheduled at any given time.  When the pool is full it will reject tasks with a Pool Full task rejection.  The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks, and the maximum pool size which allows limiting the pool so the system does not run out of resources when the pool is full.  When full, new tasks will be rejected until a thread becomes free to process new tasks.

Medium Priority Tasks

Medium priority tasks run in the medium priority pool just below the highest thread priority of the Operating System and are ordered based on task.  For example tasks of different types may run at the same time while tasks such as a system backup will run in its own queue such that the next backup can only happen after the current backup finishes.  Each medium priority task has a maximum queue size (outlined below) that when full and a new task is attempted to be run it will be rejected with a Queue Full task rejection.

The Medium priority thread pool does not have the same restriction as the High priority pool in terms of available threads to perform the work.  This pool is capable of queueing tasks if there are no free threads.  This means that in an overburdened system the Medium priority pool can grow unbounded.  When the pool is full it will queue tasks and run them when a thread becomes available.  The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks.

Low Priority Tasks

Low priority tasks run in the low priority pool at the normal thread priority of the Operating System and are un-ordered. This means that any low priority task can run independent of any other low priority task.  The tasks are supplied to the pool and will be processed in no guaranteed order.  This pool behaves like the Medium priority pool such that tasks that  have no threads to process them are queued until a thread is free.

The Low priority thread pool does not have the same restriction as the High priority pool in terms of available threads to perform the work.  This pool is capable of queueing tasks if there are no free threads.  This means that in an overburdened system the Low priority pool can grow unbounded.  When the pool is full it will queue tasks and run them when a thread becomes available.  The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks.

Configurable Priority

These tasks have a configurable priority via System Settings.  If run in the Medium or High pools they are ordered, if run in the Low pool the tasks have no order.