The OptaX Project Decentralized coflow scheduling

OPTAX: Decentralized Coflow Monitoring and Scheduling for Tiny Tasks


Overview

Task-aware flow schedulers collect task information across the data center to optimize task-level performance. However, the majority of the tasks, which generate short flows and are called tiny tasks, have been largely overlooked by current schedulers. The large number of tiny tasks brings significant overhead to the centralized schedulers, while the existing decentralized schedulers are too complex to fit in commodity switches.

To address this problem, we present OptaX, a lightweight, commodity-switch-compatible scheduling solution that efficiently monitors and schedules flows for tiny tasks with low overhead. OptaX monitors system calls and buffer footprints to recognize the tiny tasks, and assigns them with higher priorities than larger ones. The tiny tasks are then transferred in a FIFO manner by adjusting two attributes, namely, the window size and round trip time, of TCP. We have implemented OptaX as a Linux kernel module, and experiments on our 37-server testbed show that OptaX is at least 2.2x faster than fair sharing, and 1.2x faster than only assigning tiny tasks with the highest priority.

Funding

This work was supported by NSFC (61379055)

Contact us

We welcome feedback, suggestions, patches, and so forth. Send them to sdiris@gmail.com


Last modified: 22 August, 2017 /