Home / Current Issue / Paper 1701535
Hadoop MapReduce Performance Improvement In Distributed System
Subject area: Science,Engineering and Technology · Area of research: Computer Engineering
Abstract
MapReduce is currently a parallel computing framework for distributed processing of large-scale data intensive application. The most important performance metric is job execution time but it can be seriously impacted by straggler machines. Speculative execution is a common approach for this problem by backing up slow tasks on alternative machines. Some schedulers with speculative execution have been proposed but they have some weaknesses: (i) they cannot calculate the progress rate accurately because the progress scores of the phases are set to constant values which may be totally different for heterogeneous environment, (ii) they define the stragglers by specifying a static threshold value which calculates the temporal difference between an individual task and the average task progression. To get the better performance, this paper proposes an algorithm identifying the stragglers by the more accurate progress of each job based on its own historical information and using a dynamic threshold value adjusting the continuously varying environment automatically
How to cite this paper
@article{1701535,
author = {Saw Mya Nandar, Thaint Zarli Myint},
title = {Hadoop MapReduce Performance Improvement In Distributed System},
journal = {Iconic Research And Engineering Journals},
year = {2019},
volume = {3},
number = {2},
pages = {487-493},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1701535.pdf},
abstract = {MapReduce is currently a parallel computing framework for distributed processing of large-scale data intensive application. The most important performance metric is job execution time but it can be seriously impacted by straggler machines. Speculative execution is a common approach for this problem by backing up slow tasks on alternative machines. Some schedulers with speculative execution have been proposed but they have some weaknesses: (i) they cannot calculate the progress rate accurately because the progress scores of the phases are set to constant values which may be totally different for heterogeneous environment, (ii) they define the stragglers by specifying a static threshold value which calculates the temporal difference between an individual task and the average task progression. To get the better performance, this paper proposes an algorithm identifying the stragglers by the more accurate progress of each job based on its own historical information and using a dynamic threshold value adjusting the continuously varying environment automatically},
month = {August},
}