File Download
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1145/2907294.2907296
- Scopus: eid_2-s2.0-84978512005
Supplementary
-
Citations:
- Scopus: 0
- Appears in Collections:
Conference Paper: BAShuffler: maximizing network bandwidth utilization in the shuffle of YARN
Title | BAShuffler: maximizing network bandwidth utilization in the shuffle of YARN |
---|---|
Authors | |
Keywords | YARN MapReduce Shuffle Network Scheduling |
Issue Date | 2016 |
Publisher | ACM. |
Citation | The 25th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2016), Kyoto, Japan, 31 May-4 June 2016. In Conference Proceedings, 2016, p. 281-284 How to Cite? |
Abstract | YARN is a popular cluster resource management platform. It does not, however, manage the network bandwidth resources which can significantly affect the execution performance of those tasks having large volumes of data to transfer within the cluster. The shuffle phase of MapReduce jobs features many such tasks. The impact of underutilization of the network bandwidth in shuffle tasks is more pronounced if the network bandwidth capacities of the nodes in the cluster are varied. We present BAShuffler, a bandwidth-aware shuffle scheduler, that can maximize the overall network bandwidth utilization by scheduling the source nodes of the fetch flows at the application level. BAShuffler can fully utilize the net-work bandwidth capacity in a max-min fair network. The experimental results for a variety of realistic benchmarks show that BAShuffler can substantially improve the cluster's shuffle throughput and reduce the execution time of shuffle tasks as compared to the original YARN, especially in heterogeneous network bandwidth environments. |
Description | Session 8: Potpourri (Short Paper) |
Persistent Identifier | http://hdl.handle.net/10722/232186 |
ISBN |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Liang, F | - |
dc.contributor.author | Lau, FCM | - |
dc.creator | sml 161107 | - |
dc.date.accessioned | 2016-09-20T05:28:19Z | - |
dc.date.available | 2016-09-20T05:28:19Z | - |
dc.date.issued | 2016 | - |
dc.identifier.citation | The 25th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2016), Kyoto, Japan, 31 May-4 June 2016. In Conference Proceedings, 2016, p. 281-284 | - |
dc.identifier.isbn | 978-1-4503-4314-5 | - |
dc.identifier.uri | http://hdl.handle.net/10722/232186 | - |
dc.description | Session 8: Potpourri (Short Paper) | - |
dc.description.abstract | YARN is a popular cluster resource management platform. It does not, however, manage the network bandwidth resources which can significantly affect the execution performance of those tasks having large volumes of data to transfer within the cluster. The shuffle phase of MapReduce jobs features many such tasks. The impact of underutilization of the network bandwidth in shuffle tasks is more pronounced if the network bandwidth capacities of the nodes in the cluster are varied. We present BAShuffler, a bandwidth-aware shuffle scheduler, that can maximize the overall network bandwidth utilization by scheduling the source nodes of the fetch flows at the application level. BAShuffler can fully utilize the net-work bandwidth capacity in a max-min fair network. The experimental results for a variety of realistic benchmarks show that BAShuffler can substantially improve the cluster's shuffle throughput and reduce the execution time of shuffle tasks as compared to the original YARN, especially in heterogeneous network bandwidth environments. | - |
dc.language | eng | - |
dc.publisher | ACM. | - |
dc.relation.ispartof | Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2016 | - |
dc.subject | YARN | - |
dc.subject | MapReduce | - |
dc.subject | Shuffle | - |
dc.subject | Network Scheduling | - |
dc.title | BAShuffler: maximizing network bandwidth utilization in the shuffle of YARN | - |
dc.type | Conference_Paper | - |
dc.identifier.email | Lau, FCM: fcmlau@cs.hku.hk | - |
dc.identifier.authority | Lau, FCM=rp00221 | - |
dc.description.nature | link_to_OA_fulltext | - |
dc.identifier.doi | 10.1145/2907294.2907296 | - |
dc.identifier.scopus | eid_2-s2.0-84978512005 | - |
dc.identifier.hkuros | 267167 | - |
dc.identifier.spage | 281 | - |
dc.identifier.epage | 284 | - |
dc.publisher.place | United States | - |