This thesis addresses the significant challenges in optical wireless communications in space, which are adversely affected by atmospheric turbulence, light attenuation, and detector noise, leading to degraded communication reliability. To mitigate these issues, this thesis develops a neural network-based channel estimator that is optimized across a wide range of signal-to-noise ratio levels. The proposed estimator achieves performance comparable to the minimum mean square error estimator while maintaining reduced computational complexity. Additionally, a novel autoencoder (AE) framework is introduced, incorporating advanced features such as layer normalization and multiple decoders. These enhancements improve receiver learning capabilities and bit error rate (BER) performance under both perfect and imperfect channel state information (CSI) conditions.

The AE framework presented in this thesis is designed to handle multiple code rates across diverse fading channels, making it a scalable and an adaptable solution for dynamic SOC environments. Furthermore, as the Poisson channel is the most accurate channel model for optical communication, this work addresses the non-differentiability of Poisson SOC channels by integrating the covariance matrix adaptation evolution strategy with AEs, achieving near-optimal BER performance without relying on Gaussian approximations. We also propose enhanced AE designs for medium access control and transport layer settings, utilizing advanced techniques such as formulation layers to balance computational efficiency and performance.

The proposed solutions are evaluated using a system tool kit simulator for a downlink SOC channel connecting a geostationary satellite to a ground station. The results demonstrate that the NN-based channel estimator consistently outperforms state-of-the-art learning-based frameworks and achieves parity with minimum mean square error (MMSE) estimators. Similarly, the AE framework surpasses benchmark methods and popular convolutional coding techniques under both perfect and imperfect CSI conditions with various code rates. Together, the contributions of this thesis represent a significant advancement in the design of low-complexity, high-performance communication systems for space optical communications.