Enhancing Big Data Processing with a Hybrid Cloud-Edge Framework: Addressing Latency, Privacy, and Scalability Challenges in Real-Time Analytics

Abstract

This paper explores a hybrid cloud-edge computing framework for big data processing, aiming to address the limitations of traditional cloud-based systems in latency-sensitive applications like IoT and smart cities. By distributing data processing tasks across cloud and edge nodes, the framework balances the scalability and computational power of the cloud with the low-latency advantages of edge computing. Deep learning models, optimized through techniques such as pruning and quantization, enable efficient, real-time data analysis on resource-constrained edge devices. Privacy concerns, a critical aspect in this architecture, are addressed through privacy-preserving methods like federated learning and differential privacy, ensuring data protection during cloud-edge interactions. Key challenges include task allocation, resource constraints, and model adaptability, which are managed through intelligent scheduling and continuous model updates from the cloud. This approach provides a scalable, secure solution for big data applications, maximizing responsiveness and adaptability in dynamic environments. Future research will focus on enhancing resource allocation and improving model adaptability for optimized performance.