Publications -> Journal Papers

An Adaptive Efficiency-Fairness Meta-scheduler for Data-Intensive Computing


Authors: Z. Niu, S. Tang, and B. He
Title: An Adaptive Efficiency-Fairness Meta-scheduler for Data-Intensive Computing
Abstract: In data-intensive cluster computing platforms such as Hadoop YARN, efficiency and fairness are two important factors for system design and optimizations. Previous studies are either for efficiency or for fairness solely, without considering the tradeoff between efficiency and fairness. Recent studies observe that there is a tradeoff between efficiency and fairness because of resource contention between users/jobs. By leveraging the existing schedulers, a meta-scheduler is able to dynamically choose one of them for job/task scheduling at runtime. In this paper, we propose a meta-scheduler called FLEX to realize the tradeoff between system efficiency and fairness in Hadoop YARN. FLEX combines multiple existing schedulers into a single aggregated view without any modification on the original schedulers. Equipped with these candidate schedulers, FLEX utilizes machine learning approach to adaptively choose the most proper scheduler according to the characteristic of current running workload and user-defined SLA (Service Level Agreement). We implement FLEX in Hadoop YARN. We conduct experiments with real deployment in a local cluster and perform simulation studies with production traces. Experimental results show that the FLEX outperforms the state-of-the-art approach in two aspects: 1) Given a predefined threshold on the fairness loss, the FLEX reduces the makespan by up to 22% and 24% in real deployment and the large-scale simulation, respectively; 2) Given the predefined threshold on the makespan reduction, the FLEX reduces the fairness loss by up to 75% and 73% in real deployment and the large-scale simulation, respectively.
Keywords: Meta-scheduling; Efficiency-fairness tradeoff; Data-intensive; Hadoop YARN
Journal Name: IEEE Transactions on Services Computing
Publisher: IEEE
Year: 2018
Accepted PDF File: An_Adaptive_Efficiency-Fairness_Meta-scheduler_for_Data-Intensive_Computing_accepted.pdf
Permanent Link: https://doi.org/10.1109/TSC.2016.2635133
Reference: Z. Niu, S. Tang, and B. He, “An adaptive efficiency-fairness meta-scheduler for data-intensive computing,” IEEE Transactions on Services Computing, pp. 1–14, in press.
bibtex: 
@article {LILY-j58,
    author 	= {Niu, Zhaojie and Tang, Shanjiang and He, Bingsheng},
    title 	= {An Adaptive Efficiency-Fairness Meta-scheduler for Data-Intensive Computing},
    journal 	= {IEEE Transactions on Services Computing},
    year 	= {},
    month 	= {},
    volume 	= {},
    number 	= {},
    pages 	= {1-14},
    publisher 	= {IEEE},
    note		= {in press},
 }