In modern video compression and communication systems, prediction is one of the key schemes to exploit spatial and temporal redundancies. However, current approaches are suboptimal as they do not fully exploit the spatial and temporal correlations within signals. This dissertation focuses on the optimal prediction algorithms that fully utilize the correlations, and the optimal design of predictors that accounts for the rich variety of video statistics as well as the instability due to quantization error propagation in the closed-loop video coding system. Complementary to predictive coding, we also expand the design framework to the general predictive coding system, focusing on the optimal transform design that spatially de-correlates the residual data, leading to better compactness and compression performance.

The contributions in this dissertation cover the topics of spatial (intra) prediction, temporal (inter) prediction, the layered prediction in scalable coding and transform design. The contributions have been proposed to or accepted in multiple video coding standardization efforts including the Moving Picture Experts Group (MPEG) and the Alliance for Open Media (AOM), and have provided significant improvements in the video compression performance.

This dissertation is divided into two parts. The first part is concerned with developing algorithms for the compression of emerging 3D audio format, while the second part investigates optimization techniques for error-resilient predictive compression systems design.

In the first part, advances in development of compression algorithms for higher order ambisonics (HOA) data is presented. HOA has proven to be the method of choice in virtual reality applications, given its capability in reproducing spatial audio and its rendering flexibility. Recent standardization for HOA compression adopted a framework wherein HOA data are decomposed into principal components that are then encoded by standard audio coding, i.e., frequency domain quantization and entropy coding to exploit psychoacoustic redundancy. A noted shortcoming of this approach is the occasional mismatch in principal components across blocks, and the resulting suboptimal transitions in the data fed to the audio coder. In this dissertation, we propose a framework where singular value decomposition (SVD) is performed after transformation to the frequency domain via the modified discrete cosine transform (MDCT). This framework not only ensures smooth transition across blocks, but also enables frequency dependent SVD for better energy compaction. Moreover, we introduce a novel noise substitution technique to compensate for suppressed ambient energy in discarded higher order ambisonics channels, which significantly enhances the perceptual quality of the reconstructed HOA signal. In the next step, to reduce the burden of side information, a new encoding architecture is presented, where transform matrices are estimated backward-adaptively. This framework allows a more frequent usage of optimal SVD, thereby approaching the full potential of frequency

domain SVD. Also the division of HOA data into predominant and ambient components in current schemes, is difficult to perceptually optimize and ignores spatial inter channel masking effects. To address this issues, a new encoding framework for compression of HOA data is presented, where a null-space basis vector extension technique enables all compression to be performed in the SVD domain, and a jointly computed common masking threshold accounts for effects of spatial masking across components.

The second part is concerned with developing optimization techniques for error-resilient predictive compression systems design. Prediction is used in virtually all compression systems and when such a compressed signal is transmitted over unreliable networks, packet losses can lead to significant error propagation through the prediction loop. Despite this, the conventional design technique completely ignores the effect of packet losses, and estimates the prediction parameters to minimize the mean squared prediction error, and optimizes the quantizer to minimize the reconstruction error at the encoder. While some design techniques have been proposed to

accurately estimate and minimize the end-to-end distortion (EED) at the decoder

that accounts for packet losses, they operate in a closed-loop, which introduces a mismatch between statistics used for design and statistics used in operation, causing a negative impact on convergence

and stability of the design procedure. The first contribution of the dissertation is this part is proposing an effective technique for designing a compression system with a first order linear predictor, that accounts for the instability caused by error propagation due to packet losses, and enjoys stable statistics during design by employing open-loop iterations that on convergence mimic closed loop operation.

End-to-end distortion (EED) estimation, accounting for error propagation

and concealment at the decoder, has been originally developed for video coding, and enables optimal rate-distortion (RD) decisions at the encoder. However, this approach was limited to the video coder’s

simple setting of a single tap constant coefficient temporal predictor. This thesis considerably generalized the framework to account for: i) high order prediction filters, and ii) filter adaptation to local

signal statistics. We demonstrate how this EED estimate

can be leveraged, by an encoder with short and long term linear

prediction, to improve RD decisions and achieve major performance gains. The approach is further extended to estimate EED in speech coders. The error propagation problem is exacerbated in this case, as standard coders not only predict the signal from past frames, but also the parameters (in the line spectral frequency domain) employed for such prediction. Hence, the prediction loop propagates errors in the reconstructed signal as well as errors in the prediction parameters. A recursive algorithm is proposed to estimate, at the encoder, the overall EED, by the subterfuge of parallel tracking of decoder statistics for prediction parameters and signal reconstructions, in their respective domains, which are then combined to obtain the ultimate EED estimate.

This dissertation focuses on the predictive coding of video contents based on optimal prediction principles.

In the first part of the dissertation, the prediction scheme of error-resilient video coding with lossy networks is investigated. With packet-based networks suffering from potential packet loss, the prediction quality may be severely impacted not only by the loss of information, but also by the error propagation through the prediction loop. To account for such influence of lossy channels, the accurate estimation of end-to-end distortion (EED) is crucial for the encoder to perform optimal decisions. Although the recursive per-pixel optimal estimate (ROPE) and the spectral coefficientwise optimal recursive estimate (SCORE) serve as well-know solutions to optimally estimate EED in the pixel domain and the transform domain, their performance is also constrained by the incompatiblity with recent advanced coding tools. As a first step in this dissertation, the SCORE calculations are modified in order to enable the encoder to consider channel losses accurately while being able to maintain the performance improvement due to the variable block sizes. Furthermore, it is recognized that the existing tools are not designed for lossy networks, and thus a novel framework specifically tailored for this situation is proposed with a soft-reset prediction mode. With the accurate EED estimation approach of such mode established, the encoder is able to fine-tune the error propagation and thus achieves a significant performance gain. As another example, we also establish EED estimation recursions for state estimation of wireless sensor networks and proposed an adaptive approach to account for channel errors in Kalman filter, which further proves the significance of EED estimation.

The second part of the dissertation shifts focus to the bi-directional motion compensated prediction in video coding. It is first pointed out that the conventional scheme of bi-directional motion compensation is sub-optimal, since the existing motion information among the reference frames are not efficiently utilized and the block-based motion estimation is overly crude. To overcome this issue, a novel framework with the co-located reference frame (CLRF) is proposed, where a reference frame (CLRF) is interpolated by the motion field estimated between the reference frames at the decoder, without explicit transmission of such motion field. An extra step of block-based motion estimation is then performed on top of CLRF to correct possible motion offsets. Performance gains shown by experimental results prove the effectiveness of the framework.

Estimating motion field at the decoder, however, suffers from quantization error and significant complexity rise. Therefore, we then propose to apply an estimation-theory based approach to utilize the motion vectors (MVs) already available to the decoder, and treat the associated reference pixels as observations of the current block. An optimal linear estimator is then derived and used to interpolate the CLRF. With greatly reduced complexity, this approach also provides significant coding performance improvement.

The available MV candidates are then also utilized to predict the MV of the current block, in order to remove redundancy when coding MVs. Instead of a linear estimator, a novel scheme is designed to find the MV prediction that is most consistent with the MV candidates (observations) given a certain pixel correlation model. Experimental results show that the proposed scheme also achieves a boost in coding performance.

Codebook design, generation, and adaptation, based on matching to stochastic source examples or prior knowledge of source distribution, has played a central role in many applications of source coding. The original iterative ''natural type selection'' (NTS) algorithm performs stochastic codebook generation of memoryless sources, and achieves the rate-distortion bound, as it asymptotically converges to the optimal codebook reproduction distribution, Q*. However, these optimality results are subject to significant limitations that compromise the practical applicability of NTS, namely: i) the string length L is required to go to infinity at the outset, before NTS iterations begin, whereas the iteration complexity is exponential in L, and ii) it is only applicable to discrete and memoryless sources, thus precluding a vast portion of important lossy coding applications. This thesis offers means to eliminate or circumvent these critical shortcomings by proposing new and enhanced NTS algorithms, complemented by optimality proofs that are not subject to the above limitations. To circumvent the need to start with asymptotically large string length, L, the approach leverages a maximum likelihood framework to estimate, at each NTS iteration n, the reproduction distribution most likely to generate the sequence of K length-L codewords that respectively and independently “d-match” (i.e., are within distortion d from) a sequence of K length-L source words. The reproduction distribution estimated at iteration n is used to regenerate the codebook for iteration n+1. The sequence of reproduction distributions estimated by NTS is shown to converge, asymptotically in K, n, and L (in this order), to the optimal distribution that achieves the rate-distortion bound. Thus, the string length L is the last parameter to be sent to infinity. Moreover, it is established that, for finite length L, the new NTS algorithm converges to the best achievable distribution, i.e., as constrained by the string length L, and details are provided for various types of sources, where numerical simulations show that the algorithm rate of convergence in n for finite length L is at least as fast as convergence in n with infinite L. To handle sources with memory, NTS is further generalized by considering source sub-vectors or ''super-symbols'', of memory depth M, during d-match search in the codebook, maximum likelihood estimation of reproduction distribution, and codebook regeneration. Asymptotic convergence, in L and M, to the optimal reproduction distribution is also established for sources with memory. As for, perhaps the more challenging, sources over continuous alphabet spaces, which are inconsistent with the traditional concept of ''type'' or ''typical sequence'', in the proposed asymptotically optimal approach, we employ empirical probability measures for codebook reproduction distribution estimation.

Methodologies for optimal codebook generation and adaptation are further developed and employed in two promising example applications in the areas of i) wireless communications and ii) machine learning. In particular, for 5G cellular systems and next generation wireless local area networks, directional beamforming with large antenna arrays is key to mitigating the substantial signal loss experienced at the millimeter wave frequency band, where it entails a significant increase in the number of beams required to maintain cell coverage, and hence an increase in the beam management overhead necessary to maintain link with mobile users. This observation, in turn, suggests that the underlying problem of finding the optimal set of beam steering directions will benefit from fundamental signal processing and codebook design methodologies, and specifically from basic principles and algorithms for cluster analysis. This part of the thesis establishes and exploits the equivalence between the problem of optimizing a set of beam steering directions and the classical problems of clustering in pattern recognition and codebook design in data compression, albeit with an unusual distortion measure. Subsequently, a global optimization approach within the deterministic annealing framework is derived, to circumvent poor local optima that may riddle the cost surface under the classical gradient descent clustering techniques. System simulation results show that the proposed approaches deliver considerable gains, over the baseline beam steering techniques, in terms of average signal-to-noise ratio.

The third part of the thesis is concerned with codebook design and adaptation for machine learning or artificial intelligence. Machine learning applications have exploded in recent years due to the availability of huge data sets, as well as advances in computational and storage capabilities. Although successful methods have been proposed to reduce learning system complexity while maintaining required accuracy levels, theoretical understanding of the underlying trade-offs remains elusive. In this work, the classical supervised learning problem is reformulated within a rate-distortion framework. It provides insights into the underlying accuracy-complexity trade-offs, by considering the overall learning system as consisting of two components. The first is tasked with extracting (learning) from the source the minimal number of information bits necessary to ultimately achieve the prescribed output accuracy. The learned bits are then used to retrieve the desired output from the second component, an appropriately designed codebook. The premise here is that an optimal system is characterized by having to learn the minimum amount of information from the source, just sufficient to yield the system output at the desired precision, which implies efficiency in terms of system complexity, generalization and training data requirements. The design and training of such a reformulated system is detailed, and asymptotically optimal performance that achieves the rate-distortion bound is established.

Recent advances over the past two decades in neuroimaging methods have enabled us to map the connectivity of the brain. In parallel, pathophysiological models of brain disease have shifted from an emphasis on understanding pathology in specific brain regions to characterizing disruptions to interconnected neural networks. Nevertheless, these recent methods for mapping brain connectivity are still under development. Every step of the mapping process becomes a potential source for additional error due to noise or artifacts that could impact final analyses. Segmentation, parcellation, registration, and tractography are some of the steps where this occurs. Moreover, mapping the connectivity in a brain lesion is even more susceptible to errors in these steps.

In this body of work, I describe multiple new methods for improving the accuracy of mapping lesion connectivity by reducing errors at the tractography stage which is the most error prone stage. First, we develop an approach for directly normalizing streamlines into a template space that avoids performing tractography in the normalized template space, reducing the error of connectomes constructed in the template space with respect to the ground truth native space connectome. Second, we develop a rapid approach for performing shortest path tractography and constructing shortest path probability weighted connectomes which increases the connection specificity relative to local streamline tracking approaches. We then demonstrate how our shortest path tractography approach can be used construct a disconnectome, a connectivity map of the proportion of connections lost due to intersecting a lesion. We then develop a fast, greedy graph-theoretic algorithm that extracts the maximally disconnected subgraph containing brain regions with the greatest shared loss of connectivity. Finally, we demonstrate how combining methods from diffusion based image inpainting and optimal estimation can be used to restore or inpaint corrupted fiber diffusion models in lesioned white matter tissue, enabling tractography and the study of lesion connectivity and modeling of microstructural measures in the patient’s native space.

Modern advances in imaging technology have enabled the collection of huge amounts of multimodal imagery of complex biological systems. The extraction of information from this data and subsequent analysis are essential in understanding the architecture and dynamics of these systems. Due to the sheer volume of the data, manual annotation and analysis is usually infeasible, and robust automated techniques are the need of the hour. In this dissertation, we present three hidden Markov model (HMM)-based methods for automated analysis of multimodal biomedical images. First, we outline a novel approach to simultaneously classify and segment multiple cells of different classes in multi-biomarker images. A 2D HMM is set up on the superpixel lattice obtained from the input image. Parameters ensuring spatial consistency of labels and high confidence in local class selection are embedded in the HMM framework, and learnt with the objective of maximizing discrimination between classes. Optimal labels are inferred using the HMM, and are aggregated to obtain global multiple object segmentation. We then address the problem of automated spatial alignment of images from different modalities. We propose a probabilistic framework, constructed using a 2D HMM, for deformable registration of multimodal images. The HMM is tailored to capture deformation via state transitions, and modality-specific representation via class-conditional emission probabilities. The latter aspect is premised on the realization that different modalities may provide very different representation for a given class of objects. Parameters of the HMM are learned from data, and hence the method is applicable to a wide array of datasets. In the final part of the dissertation, we describe a method for automated segmentation and subsequent tracking of cells in a challenging target image modality, wherein useful information from a complementary (source) modality is effectively utilized to assist segmentation. Labels are estimated in the source domain, and then transferred to generate preliminary segmentations in the target domain. A 1D HMM-based algorithm is used to refine segmentation boundaries in the target image, and subsequently track cells through a 3D image stack. This dissertation details techniques for classification, segmentation and registration, that together form a comprehensive system for automated analysis of multimodal biomedical datasets.

Virtual reality's resurgence has intensified interest in higher order Ambisonics (HOA), renowned for its ability to recreate spatial audio across diverse speaker setups which is crucial for many applications. Using spherical harmonics to create a 3D soundfield, HOA's popularity in spatial audio storage and transmission is significant. However, challenges arise in enabling immersive experiences due to the potential for HOA to encompass up to 64 audio channels, necessitating effective compression methods. This thesis addresses this challenge by proposing new and tailored algorithms for the compression and loss concealment of HOA signals. The first part of the thesis discusses a new adaptive framework for HOA compression which considers both the previous frame reconstructed data and the current frame data, to obtain a more relevant set of SVD basis vectors spanning the null space, in order to extend the available set of dominant basis vectors, at the decoder, at little bitrate cost, leading to significantly improved audio quality. The second part of the thesis focuses on low-delay HOA compression. Modern codecs use a combination of inter-channel and inter-frame linear predictors or combine frame-based singular value decomposition (SVD) with the MDCT. This thesis shows that reduced delay and superior bitrate can be gained, by instead applying an adaptive SVD transform, which relies on previously decoded data rather than the current time samples, for inter-channel decorrelation, as well as LPC and cascaded long term prediction (which can capture the periodic components of polyphonic signals) to capture short-term and long-term temporal correlations, respectively. The third part of the thesis focuses on loss concealment for HOA. Current methods for loss concealment involve essentially applying a predictor trained on past and future data to predict the lost frame. However, such methods do not consider the spatial aspects of HOA. The thesis shows how significant improvements can be made by decorrelating the signal using SVD, treating the audio aspect with a predictor and the spatial basis vectors with interpolation on a sample-by-sample basis, before recombining the audio and spatial aspects of the signal to arrive at a superior estimate of the lost frame.

The main focus of this dissertation is on the optimal design of motion compensation scheme for predictive coding in video compression. The “sub-optimality” of conventional block-based motion compensation scheme, wherein the impact of a motion vector is confined within in a rigid rectangular block, motivates our design of a multi-hypothesis motion compensation scheme. We explicitly treat motion vectors as pointers to observation sources. Given the high-correlation between adjacent pixels in nature images, the motion vectors of neighboring blocks can point to relevant estimates of the current target. This dissertation work builds on this paradigm and demonstrates advanced techniques devised by incorporating the information of the entire motion vector field.

We first directly formulate the problem of motion compensation with multiple estimates as a linear estimation problem, and design a training method to derive the optimal linear coefficients to avoid overfitting as well as to minimize the ultimate reconstruction errors rather than prediction errors. As a single set of coefficients cannot capture the varying statistics of video sequences, we design K sets of coefficients which are trained off-line through “K-mode” iterative clustering techniques. By switching between the predefined sets of coefficients, the encoder can adapt to local statistics. This approach is then extended to the setting of variable block size partitioning to enjoy the substantial gain provided by the flexibility of dividing blocks to approximate object shapes. As the additional side information to indicate the set of prediction coefficients used to generate the final prediction is generally not negligible, a parametric framework is proposed to model the statistics of estimates and target pixels. The model leverages the first-order Markov property for image signals and relationships between motion vectors in the motion field. As a result, the coefficients derived from the model can automatically adapt to local variations without additional side information. Moreover, using the parametric approach, we can combine estimates from any number of motion vectors at any pixel location, which allows us to completely break free from the block structure by allowing a motion vector’s influence to be of arbitrary shape.

The reminder of this dissertation is focused on optimization on the AV1 encoder, the open source video encoder founded by the Alliance of Open Media (AOM). We first introduce a new coding tool to extend the number of reference frames, and then we use the existing coding tools to design a coding structure, wherein the reference frames are allocated to cover a wider temporal range to offer more diversities. This new design allows the encoder to better capture temporal variations. Finally, we introduce a complementary compound prediction mode, which is designed and optimized for the blocks where the existing compound modes fall short. Simulations provide experimental evidences for the efficiency of proposed mode with consistent coding gain across all bit-rate range.

This dissertation is concerned with the problem of global optimization of delay constrained communication and control strategies. Specifically, the objective is to obtain optimal encoder and decoder functions that map between the source space and the channel space, to minimize a given cost functional. The cost surfaces associated with these problems are highly complex and riddled with local minima, rendering gradient descent based methods ineffective. This thesis proposes and develops a powerful non-convex optimization method based on the concept of deterministic annealing (DA) - which is derived from information theoretic principles with analogies to statistical physics, and was successfully employed in several problems including vector quantization, classification and regression. DA has several useful properties including reduced sensitivity to initialization and strong potential to avoid poor local minima. DA-based optimization methods are developed here for the following fundamental communication problems: the Wyner-Ziv setting where only a decoder has access to side information, the distributed setting where independent encoders transmit over independent channels to a central decoder, and analog multiple descriptions setting which is an extension of the well known source coding problem of multiple descriptions. Comparative numerical results are presented, which show strict superiority of the proposed method over gradient descent based optimization methods as well as prior approaches in literature. Detailed analysis of the highly non-trivial structure of obtained mappings is provided.

The thesis further studies the related problem of global optimization of controller mappings in decentralized stochastic control problems, including Witsenhausen's celebrated 1968 counter-example. It is well-known that most decentralized control problems do not admit closed-form solutions and require numerical optimization. An optimization method is developed, based on DA, for a class of decentralized stochastic control problems. Comparative numerical results are presented for two test problems that show strict superiority of the proposed method over prior approaches in literature, and analyze the structure of obtained controller functions.

The focus of the dissertation is on optimal prediction paradigms and the complementary design of transforms for video compression. One main line of research is on motion compensated prediction for spherical videos. Standard approaches project spherical videos onto planes for processing with traditional 2D video coding standards. Such approaches are significantly sub-optimal as standard video coders only allow for block translations in the critical tool of motion compensated prediction, which is incompatible with the expected motion in projected spherical video. Specifically, the effective sampling density varies over the sphere and the resulting locally varying warping yields complex non-linear motion in the projected domain. Moreover, motion vector in the projected domain does not have a useful physical interpretation. To address these shortcomings, the thesis presents a rotational motion model that performs motion compensation in terms of rotations along geodesics on the sphere. Rotation preserves object shape and size on the sphere. A motion vector in this model implicitly specifies an axis of rotation and the degree of rotation about that axis, to convey the actual motion of objects on the sphere. Complementary to the novel motion model, an effective motion search technique that is tailored to the sphere’s geometry is presented for improved motion estimation.

The thesis then considers an important class of spherical videos whose dynamics involve camera motion. The thesis presents a new geodesic translation motion model that captures the motion field on the sphere, and capitalizes on insights into the perceived motion on the sphere due to camera translation. Specifically, surrounding static points are perceived as moving along their respective geodesics, which all intersect at the poles corresponding to the camera velocity axis. The method further exploits insights into the displacement rate of static points, which depends on object depth and degree of elevation on the sphere (with respect to the camera velocity axis). Complementary to the new motion model, a search grid tailored to capture expected geodesic motion on the sphere for effective motion estimation is presented.

Another focus related to predictor optimization is on design of prediction filters for adaptive compression of non-stationary signals with applications to video coding. The design poses several challenges including: i) catastrophic instability due to statistical mismatch driven by error propagation through the prediction loop, and ii) severe non-convexity of the cost surface that is often riddled with poor local minima. Motivated by these challenges, the thesis presents a near-optimal method for designing prediction modes for adaptive compression. The design builds on a stable, open-loop platform, but with a subterfuge that ensures that it is asymptotically optimized for closed-loop operation. The non-convexity is handled by deterministic annealing, a powerful optimization tool to avoid poor local minima. The impact of the design paradigm on practical applications is demonstrated by designing temporal prediction filters in video coding.

The second line of research focuses on offline design of transforms for inter-prediction residuals, a complementary step to the effective prediction paradigms. Existing codecs for regular 2D videos use standard trigonometric transforms. These transforms are only optimal under certain assumptions that are highly questionable for inter-prediction residue. For projected spherical videos, derivation of transforms even under certain assumptions is a highly challenging task. Thus, there is a strong motivation for a data-driven approach to learn these transforms. The joint design of multiple transform modes is highly challenging due to critical stability problems inherent to feedback through the codec’s prediction loop, wherein training updates inadvertently impact the signal statistics the transform ultimately operates on, and are often counter-productive. It is the premise of this work that a truly effective switched transform design procedure must account for and circumvent this shortcoming. We introduce a data-driven approach to design optimal transform modes for adaptive switching by the encoder. Most importantly, to overcome the critical stability issues, the approach is derived within an asymptotic closed loop (ACL) design framework. The design yields transforms that outperform the transforms obtained by standard design procedures.