Machine learning applications have widely expanded from single, isolated systems to complex distributed environments, such as mobile computing, sensor networks, and healthcare systems. These environments often include thousands of nodes (e.g., edge devices) that collect data and perform local training or inference, exhibiting considerable heterogeneity in data and computing resources. The heterogeneity poses great challenges for training machine learning models across distributed systems.

This dissertation discusses practical scenarios for deploying machine learning in distributed environments. We introduce new methods for effective learning under heterogeneity, focusing on three dimensions: robustness, scalability, and quality. In the first part, we address data heterogeneity by leveraging contextual cues to align and adapt models, ensuring robustness to variations in distributed training data. In the second part, we manage system heterogeneity by developing knowledge aggregation methods that enable nodes with varying capacities to collaborate inclusively and efficiently. In the third part, we develop methods for deriving implicit contextual information from data, which is essential for finding correlations among distributed domains and enhancing model quality. The proposed approaches are designed to be model-agnostic, supporting various applications and system configurations. Extensive evaluations demonstrate that our methods achieve state-of-the-art performance across a wide range of real-world applications, including image and language processing, human sensing and mobile computing tasks, such as healthcare and activity recognition. By addressing different heterogeneity scenarios, our methods improve the global model trained on heterogeneous data sources by 6.14%, enhance node-specific adapted models by up to 14.85%, boost performance across nodes with diverse capacities, and accelerate training by 12×. The contributions in this dissertation enhance the practicality of Artificial Intelligence (AI) at the edge, facilitating the implementation of ubiquitous intelligent systems for seamless assistance.