Intelligent Load Balancing Framework for Distributed Big Data Processing System

This paper proposes an intelligent load balancing framework for distributed big data processing systems that integrates machine learning techniques with adaptive weight-based decision mechanisms. The study addresses limitations of traditional static load balancing methods, which do not account for dynamic workload variations and heterogeneous request characteristics, leading to inefficient resource utilization and bottlenecks in multi-node environments. The proposed approach combines an online learning model for real-time estimation of request complexity with multi-parameter evaluation of node states, including CPU utilization, memory consumption, queue length, response latency, and cache efficiency. A dynamic weighting strategy is used to construct an integrated load indicator for adaptive request distribution across nodes. The framework is deployed within a multi-layer distributed architecture consisting of clustered application servers, distributed databases, caching subsystems, and monitoring components, ensuring scalable and fault-tolerant processing. For evaluation, a three-node simulation environment was used with 10,000 heterogeneous requests, followed by extended testing on semi-realistic workload traces derived from web traffic patterns and database query logs. The dataset included over 1.2 million requests, capturing bursty arrivals, skewed distributions, and heterogeneous complexity. Experimental results show that the proposed method improves load distribution uniformity to 6%, reduces average response time to 210 ms, and increases throughput up to 13,800 requests per second. Statistical validation using confidence intervals and hypothesis testing confirms a 47% (±3.2% at 95% confidence level) reduction in mean response time and throughput improvement up to 14,200 requests per second under realistic workloads.