File Download
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1109/IPDPS.2015.17
- Scopus: eid_2-s2.0-84971426757
- WOS: WOS:000380545200028
- Find via
Supplementary
- Citations:
- Appears in Collections:
Conference Paper: SMapReduce: optimising resource allocation by managing working slots at runtime
Title | SMapReduce: optimising resource allocation by managing working slots at runtime |
---|---|
Authors | |
Keywords | Resource Management MapReduce Hadoop YARN Performance Modeling |
Issue Date | 2015 |
Publisher | IEEE Computer Society. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000530 |
Citation | The 29th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2015), Hyderabad, India, 25-29 May 2015. In IPDPS Proceedings, 2015, p. 281-190 How to Cite? |
Abstract | Hadoop version 1 (HadoopV1) and version 2 (YARN) manage the resources in a distributed system in different ways. HadoopV1 executes MapReduce tasks in working slots that are statically configured, YARN uses a set of task containers to encapsulate its memory and CPU resources. However, neither of them considers the runtime performance of the cluster when deciding the proper number of concurrent tasks to run on each node to achieve the optimal throughput. In order to gain higher performance, the users of Hadoop usually need to use their experience to carefully configure the resources of the cluster and the resources needed by their jobs. But as the workload is typically always changing in the cluster, rarely could such a manual configuration lead to optimized performance. In this paper, we study the MapReduce job performance in HadoopV1 and YARN with different resource configurations, and model the cluster throughput in terms of the resource capacity of the cluster. We propose SMapReduce, which can dynamically manage a proper number of concurrent tasks running on each node. SMapReduce can gain the maximum job throughput by considering the thrashing phenomenon and the balancing between map and reduce tasks. Evaluation results show that SMapReduce can yield significant performance speedup comparing to both HadoopV1 and YARN for various MapReduce workloads. |
Description | Open Access |
Persistent Identifier | http://hdl.handle.net/10722/219225 |
ISSN | |
ISI Accession Number ID |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Liang, F | - |
dc.contributor.author | Lau, FCM | - |
dc.date.accessioned | 2015-09-18T07:18:11Z | - |
dc.date.available | 2015-09-18T07:18:11Z | - |
dc.date.issued | 2015 | - |
dc.identifier.citation | The 29th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2015), Hyderabad, India, 25-29 May 2015. In IPDPS Proceedings, 2015, p. 281-190 | - |
dc.identifier.issn | 1530-2075 | - |
dc.identifier.uri | http://hdl.handle.net/10722/219225 | - |
dc.description | Open Access | - |
dc.description.abstract | Hadoop version 1 (HadoopV1) and version 2 (YARN) manage the resources in a distributed system in different ways. HadoopV1 executes MapReduce tasks in working slots that are statically configured, YARN uses a set of task containers to encapsulate its memory and CPU resources. However, neither of them considers the runtime performance of the cluster when deciding the proper number of concurrent tasks to run on each node to achieve the optimal throughput. In order to gain higher performance, the users of Hadoop usually need to use their experience to carefully configure the resources of the cluster and the resources needed by their jobs. But as the workload is typically always changing in the cluster, rarely could such a manual configuration lead to optimized performance. In this paper, we study the MapReduce job performance in HadoopV1 and YARN with different resource configurations, and model the cluster throughput in terms of the resource capacity of the cluster. We propose SMapReduce, which can dynamically manage a proper number of concurrent tasks running on each node. SMapReduce can gain the maximum job throughput by considering the thrashing phenomenon and the balancing between map and reduce tasks. Evaluation results show that SMapReduce can yield significant performance speedup comparing to both HadoopV1 and YARN for various MapReduce workloads. | - |
dc.language | eng | - |
dc.publisher | IEEE Computer Society. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000530 | - |
dc.relation.ispartof | IPDPS Proceedings | - |
dc.rights | IPDPS Proceedings. Copyright © IEEE Computer Society. | - |
dc.rights | ©2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. | - |
dc.subject | Resource Management | - |
dc.subject | MapReduce | - |
dc.subject | Hadoop | - |
dc.subject | YARN | - |
dc.subject | Performance Modeling | - |
dc.title | SMapReduce: optimising resource allocation by managing working slots at runtime | - |
dc.type | Conference_Paper | - |
dc.identifier.email | Lau, FCM: fcmlau@cs.hku.hk | - |
dc.identifier.authority | Lau, FCM=rp00221 | - |
dc.description.nature | link_to_OA_fulltext | - |
dc.identifier.doi | 10.1109/IPDPS.2015.17 | - |
dc.identifier.scopus | eid_2-s2.0-84971426757 | - |
dc.identifier.hkuros | 253762 | - |
dc.identifier.spage | 281 | - |
dc.identifier.epage | 190 | - |
dc.identifier.isi | WOS:000380545200028 | - |
dc.publisher.place | United States | - |
dc.customcontrol.immutable | sml 151106 | - |
dc.identifier.issnl | 1530-2075 | - |