UC San Diego

Kernel Alignment has been developed and analysed in the field of multiple kernel learning in the past few decades. Recent studies have shown that kernel alignment with linear kernels can be used to measure the similarity between two sets of high-dimensional vector representations produced by neural networks. Given its theoretical guarantees in learning and practical implication in analysing machine learning models, this thesis examines linear kernel alignment and the algorithm for learning kernel alignment in domain generalisation mainly in three aspects. Firstly, with the help of sketching techniques, we demonstrate that linear kernel alignment is a decent proxy for transferability, which is defined as a score that implies how well a pretrained model would perform on a downstream task. Secondly, in the setting of transferring learnt knowledge from a pretrained neural network to a downstream task, instead of finetuning the top layer or the whole network to adapt to the new task, we propose to accumulate feature vectors from a certain number of layers for making final predictions. The layer selection is done through an algorithm for seeking an optimal convex combination of linear kernels from individual layers. The optimised combination gives as good performance as combining all layers, and the performance is better than simply finetuning the top layer of a pretrained neural network. Thirdly, in ensemble learning, by using the algorithm for learning kernels with linear kernels constructed from individual predictors, the optimal convex combination drastically prunes the predictors required for the inference whilst boosting the performance of ensemble learning methods when all predictors are used. Through these three chapters, we demonstrate the simplicity and practicality of linear kernel alignment in domain generalisation.