Advancements in Grid Computing for High-Throughput Data Processing

Abstract

Rapid growth in the volume and complexity of data-intensive tasks has led to significant interest in harnessing distributed computational resources for high-throughput data processing. Grid computing has emerged as a powerful paradigm to meet these challenges, offering a robust framework for sharing, coordinating, and aggregating heterogeneous and geographically dispersed resources. Over the last two decades, advancements in Grid computing infrastructures, middleware, scheduling algorithms, and data management techniques have significantly improved the throughput and responsiveness of large-scale scientific and commercial workflows. This paper presents a comprehensive review of the state-of-the-art in Grid computing for high-throughput data processing, highlighting key architectural developments, middleware enhancements, resource provisioning mechanisms, and novel optimization techniques for data distribution and scheduling. The paper also discusses the integration of emerging technologies such as containerization, edge computing, and machine learning-based decision-making to further elevate the efficiency and scalability of Grid computing. Finally, we provide an in-depth analysis of current challenges, open research questions, and future directions for Grid computing systems with a focus on achieving enhanced performance, reliability, and sustainability in high-throughput data processing environments.