In this thesis, we focused on investigating novelty modules integrated into popular detection network for assisting it to learn attentive representations for several practical applications in objects detection and instance segmentation tasks, including universal object detection and 3D medical image segmentation tasks. For universal object detection task, despite increasing efforts on universal representations for visual recognition, few have addressed object detection. In this thesis, we develop an effective and efficient universal object detection system that is capable of working on various image domains, from human faces and traffic signs to medical CT images. Unlike multi-domain models, this universal model does not require prior knowledge of the domain of interest. This is achieved by the introduction of a new family of adaptation layers, based on the principles of squeeze and excitation, and a new domain-attention mechanism. In the proposed universal detector, all parameters and computations are shared across domains, and a single network processes all domains all the time. Experiments, on a newly established universal object detection benchmark of 11 diverse datasets, show that the proposed detector outperforms a bank of individual detectors, a multi-domain detector, and a baseline universal detector, with a 1.3 parameter increase over a single-domain baseline detector. For 3D medical images segmentation tasks, although high resolution 3D medical images offer abundant detail information of human body parts and allow early detection of small lesions, due to the limitation of GPU memory, most methods either use down-sampled 3D volume as input, which significantly affects the detectability of small lesions, or use 2.5D networks to crop out neighboring image slices at original resolution, which loses context information along z direction. Both ways can significantly affect the performance of final model. In this paper, we propose a cross-slice spatial and channel attention module, which can maintain spatial resolution of input data, and effectively utilize context information along z direction of 3D volume. In order to get higher quality mask prediction, a cascade mask refinement module is designed to provide an objectiveness pixel-wise attention map for input feature maps. Furthermore, our scheme allows us to utilize the pretrained 2D detection models to achieve good results even with limited amount of training data, which is often met in medical applications and imposes big challenge to many deep learning methods. By utilizing the two novel modules, we achieve state-of-art performance 74.10 dice per case on Liver Tumor Segmentation Challenge(LiTS), which outperforms previous year challenge winner by 6.7 points and rank as 1st on leader board of LiTS benchmark upon submission of this paper.

Deep learning models have become the backbone of modern computer vision systems, achieving striking success in a wide range of tasks from image to video understanding. While the benchmark performance of deep neural networks never appears to saturate as they are fed more data and compute, real-world applications of these models often fall short of expectations. This is due to the presence of biases in the data, which misrepresent the underlying distribution and lead to poor generalization and discriminatory outcomes. In this thesis, we investigate the problem of bias in vision and multimodal learning systems, proposing methods to identify, measure, and mitigate biases in both the data and the models.

The first part of the dissertation introduces a new form of bias, representation bias, which measures the extend to which ground-truth labels can be inferred from spurious features in the data. We apply the concept to study the static bias in video action recognition, and propose a procedure, RESOUND, to guide the collection of datasets free from representation bias. We further develop a debiasing method, REPAIR, to mitigate the bias in existing datasets, and show that debiasing the training data can improve the generalization of video models on new datasets.

The second part shifts the focus to model-centric approaches that alleviate the vulnerability of models to data biases. We propose dynamic representation learning, a novel framework for quantifying and minimizing the static bias of video classification models, and use this to study the impact of bias removal on the transferability of video representations. We then extend the debiasing to multimodal learning, proposing a sparse video-text transformer for efficient modeling over long clips, and a training curriculum that enables temporal learning beyond static visual features.

In the final part of the dissertation, we advocate for a holistic view of bias mitigation that considers dataset and model biases jointly, and apply this strategy to improve the fairness of vision-language foundation models using generated counterfactuals. We demonstrate the benefit of this approach in unifying bias mitigation for diverse tasks and domains, and discuss the potential of holistic debiasing in future research.

Purpose

Methods

Results

Conclusions