Queue Orchestration Using an In-Memory Broker

Existing solutions for a task queue have overheads that slow down the task queueing process. The system proposed is built with the help of Redis as the message broker. The Redis Zset data structure reduces the time taken for queuing significantly relative to existing solutions such as Celery which has a high overhead due to the various other tasks that it must perform. Celery will be used as the benchmark for comparison against the proposed system’s performance. The time taken for the task in the queue will be calculated and compared for both Celery and the proposed system for 1, 2, 4, and 8 workers. From the results, it can be observed that the tasks in the proposed system spent less time in the queue for all four cases when compared with the relative performance of Celery’s, which has increasing queue time with an increase in the number of workers.