A novel scalable deep ensemble learning framework for big data classification via MapReduce integration

Abstract

Big data classification involves the systematic sorting and analysis of extensive datasets that are aggregated from a variety of sources. These datasets may include but are not limited to, electronic records, digital imaging, genetic information sequences, transactional data, research outputs, and data streams from wearable technologies and connected devices. This paper introduces the scalable deep ensemble learning framework for big data classification (SDELF-BDC), a novel methodology tailored for the classification of large-scale data. At its core, SDELF-BDC leverages a Hadoop-based map-reduce framework for feature selection, significantly reducing feature-length and enhancing computational efficiency. The methodology is further augmented by a deep ensemble model that judiciously applies a variety of deep learning classifiers based on data characteristics, thereby ensuring optimal performance. Each classifier's output undergoes a rigorous optimization-based ensemble approach for refinement, utilizing a sophisticated algorithm. The result is a robust classification system that excels in predictive accuracy while maintaining scalability and responsiveness to the dynamic requirements of big data environments. Through a strategic combination of classifiers and an innovative reduction phase, SDELF-BDC emerges as a comprehensive solution for big data classification challenges, setting new benchmarks for predictive analytics in diverse and data-intensive domains.