Publications -> Conference Papers

Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing


Authors: Z. Niu, S. Tang, and B. He
Title: Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing
Abstract: In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are two important factors for system design and optimizations. Many previous studies are either for performance or for fairness solely, without considering the tradeoff between performance and fairness. Recent studies observe that there is a trade-off between performance and fairness because of resource contention between users/jobs. However, their scheduling algorithms for bi-criteria optimization between performance and fairness are static, without considering the impact of different workload characteristics on the tradeoff between performance and fairness. In this paper, we propose an adaptive scheduler called Gemini for Hadoop YARN. We first develop a model with the regression approach to estimate the performance improvement and the fairness loss under the sharing computation compared to the exclusive non-sharing scenario. Next, we leverage the model to guide the resource allocation for pending tasks to optimize the performance of the cluster given the user-defined fairness level. Instead of using a static scheduling policy, Gemini adaptively decides the proper scheduling policy according to the current running workload. We implement Gemini in Hadoop YARN. Experimental results show that Gemini outperforms the state-of-the-art approach in two aspects. 1) For the same fairness loss, Gemini improves the performance by up to 225% and 200% in real deployment and the large-scale simulation, respectively; 2) For the same performance improvement, Gemini reduces the fairness loss up to 70% and 62.5% in real deployment and the large-scale simulation, respectively.
Keywords: Tradeoff; Data-intensive; Fairness; Performance
Conference Name: 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom'15)
Location: Vancouver, Canada
Publisher: IEEE
Year: 2015
Accepted PDF File: Gemini_An_Adaptive_Performance-Fairness_Scheduler_for_Data-Intensive_Cluster_Computing_accepted.pdf
Permanent Link: https://doi.org/10.1109/CloudCom.2015.52
Reference: Z. Niu, S. Tang, and B. He, “Gemini: An adaptive performance-fairness scheduler for data-intensive cluster computing,” in Proceedings of the 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom’15). IEEE, November 2015, pp. 66–73.
bibtex: 
@inproceedings{LILY-c52, 
   author = {Niu, Zhaojie and Tang, Shanjiang and He, Bingsheng},
   title  = {Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing},  
   booktitle = {Proceedings of the 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom'15)}, 
   year  = {2015}, 
   month = {November}, 
   pages = {66-73}, 
   location = {Vancouver, Canada},
   publisher = {IEEE},
}