Automatic Internal Parallelism Reconfiguration on Heterogeneous Low-Power Hadoop Clusters

Apache Hadoop has been widely adopted for massive data processing and analysis for over a decade, using powerful, energy-consuming server clusters. Recently, energy efficiency and capital costs have become primary concerns of data centers; hence, low-power computers, such as single board computer (SBC) clusters, have been of interest as practical alternatives. These low-cost, low-power SBC clusters on the Hadoop platform introduce new problems due to the limited computational capability. The heterogeneity of SBC clusters is inevitable after adding newer nodes or replacing older or inactive nodes, creating more challenges because the native Hadoop framework does not carefully consider computational discrepancies in each node. This paper redesigns the Yet Another Resource Negotiator (YARN) architecture with Hadoop using intranode parallelism effectively. Unlike the original Hadoop YARN, the proposed design automatically reconfigures the number of concurrently processable tasks (i.e., concurrent containers) based on the actual computing resource information each node provides. Thus, this framework judiciously applies node-level parallelism in Hadoop according to workload characteristics. In addition, the redesigned YARN provides effective Hadoop task distribution policies, particularly for heterogeneous, low-power SBC clusters. The extensive experiments with Hadoop benchmarks demonstrate that the redesigned YARN framework performs better than the original framework by an average of 15% under I/O-intensive workloads and 6% under CPU-intensive workloads.