Convolutional neural networks have been widely used in machine learning and computer vision tasks for either discriminative purposes or generative modeling.

This thesis first presents the discriminative power of CNNs for instance segmentation task, with a focus on the intrinsic challenge of the problem —- the presence of a quotient space (swapping the labels of different instances leads to the same result). We propose a simple segmentation based framework as well as three instance labeling transformation methods, namely pixel-based affinity mapping, superpixel-based affinity learning, and boundary-based component segmentation. Our methods are object proposal- and object detection- free, and achieves competitive results on various benchmark datasets.

Secondly, this thesis presents the introspective classification with convolutional neural networks that emphasizes the importance of having convolutional neural networks empowered with generative capabilities. We propose the reclassification-by-synthesis algorithm to perform training using a formulation stemmed from the Bayes theory. The single CNN classifier learned is at the same time generative -- being able to directly synthesize new samples within its own discriminative model. We conduct experiments on benchmark datasets using state-of-the-art CNN architectures, and observe improved classification results.

Finally, this thesis applies introspective learning in unsupervised vision tasks. We develop a generative model by progressively learning a sequence of CNN classifiers. The resulting generator is additionally a discriminator, being able to self-evaluate the difference between its generated samples and the given training data. In the experiments, we observe encouraging results on a number of applications including face modeling, texture modeling, unsupervised feature learning and image-to-image translation.

With the support of big-data and big-compute, deep learning has reshaped the landscape of research and applications in artificial intelligence. Whilst traditional hand-guided feature engineering in many cases is simplified, the deep network architectures become increasingly more complex. A central question is whether we can distill the minimal set of structural priors that can provide us the maximal flexibility, and lead us to richer sets of structural primitives. Those structural priors will make the learning process more effective, and potentially lay the foundations towards the ultimate goal of building general intelligent systems.

This dissertation focuses on how we can tackle different real world problems in computer vision and machine learning with carefully designed neural network architectures, guided by simple yet effective structural priors. In particular, this thesis focuses on two structural priors that have proven to be useful and generalizable in many different scenarios: the multi-scale prior, with an application in edge detection, and the sparse-connectivity prior implemented for generic visual recognition. Examples will be presented in the last part, on how to learn meaningful structures directly from data, rather than hard-wiring them by, for example, learning a convolutional pseudo-prior in the label space, or adopting a dynamic self-attention mechanism.

This thesis presents a deep neural network model that augments an existing semantic

image segmentation model with optical flow data to improve segmentation performance

on video sequences. Three network topologies combining optical flow data

layers with RGB data layers are compared. The best performing model, FlowSegA,

achieves an average per-class accuracy of 72.696% on the SegNet test set. This

is an improvement of 4.8 percentage-points versus SegNet, the RGB-only segmentation

model on which FlowSeg-A is based. The main accuracy improvements come

from the classes SignSymbol (15.4% improvement), Bicyclist(10.2%), and Pole

(9.0%). These accuracy improvements are achieved with only 1,152 (0.005%) more

parameters, and FlowSeg-A achieves this performance using the same training set and

training schedule as the SegNet algorithm.

Visual perception is a fundamental task of computer vision. Subtasks within perception can be decomposed into two types: reasoning about the generative process of images or phenomena themselves (i.e., a prior on the sensory input we expect to perceive) as well as discriminating high-level structural information contained within them. In this dissertation, we first explore how this decomposition is not as decoupled as it might appear. In particular, we show how the act of discrimination is sufficient to produce a model of generative ability. Furthermore, while vision algorithms are required to discriminate about visible aspects of a scene, it is often useful to reason about what cannot be seen within a 2D observation. We explore a practical method for eliciting so-called 2.1D information on two popular, large-scale datasets and demonstrate how this generative information can improve certain scene-level segmentation tasks. Next, we explicitly build a model of scene layout which relies on an amodal understanding (both what can and cannot be seen) of so-called "stuff" (background classes) to accurately place "things'" (objects) in a scene. Finally, we revisit the de facto method of instance segmentation within the modern age of computer vision — binary mask prediction — and question whether other approaches (i.e., boundary-based regression) can be suitable alternatives. We validate for the first time that continuous, boundary-based regression can match mask-based prediction with respect to a variety of notions of parity alongside numerous advantages in differentiability and sparsity compared to it. We believe this opens up further research into the segmentation algorithms of the future.

The purpose of this thesis is to present a novel method of learning to generate an image sequence from input a single image without sequentially aligned data. Given examples of a visual phenomena that can be divided into discrete time steps, the problem is to learn a model that takes an input from any such time and realizes this input at all other time steps in the sequence. For example, given a scenery picture in spring and output the corresponding pictures in sequence of summer, fall, and winter without changing overall layout and semantic information presented in the input picture. Furthermore, it is assumed that ground-truth aligned sequences is not provided. This broadens the real-world application of this method as it is often difficult to collect aligned sequential data for many problems. This task generalizes the unpaired image-to-image problem from generating pairs to generating sequences and associates a direction of time with the phenomena observed.

We show that this problem can be solved by incorporating Generative Adversarial Networks (GAN), a popular deep unsupervised learning technique, and a periodic assumption about the sequential visual phenomena modeled. The periodic assumption is enforced in model training by a novel Loop Consistency loss, inspired by the popular Cycle Consistency loss that has achieved huge success in unpaired image-to-image transformation. The two parts of the model, GAN and Loop Consistency, can be seen as two levels of constraints that together facilitate model training. The transformation unit itself is a neural network. We show the effects of different network architecture changes on generation quality and present the results of the model in comparison with several competitive baseline models.

Neural networks have been shown effective at learning rich low-dimensional representations of high-dimensional data such as images and text. There has also been many recent works using neural networks to learn a common embedding between data of different modes, specifically between images and textual descriptions, a task commonly referred to as learning visual-semantic embeddings. This is typically achieved using a separate encoder for images and text and a contrastive loss. Inspired by recent works in relational reasoning and graph neural networks, this work studies the effects of using a relational inductive bias on the quality of learned visual-semantic embeddings. Training and evaluation is done using caption-to-image and image-to-caption retrieval on the MS-COCO dataset.

We tackle the problem of point cloud recognition.

Unlike previous approaches where a point cloud is either converted into a volume/image or represented independently in a permutation-invariant set, we develop a new representation by adopting the concept of shape context as the building block in our network design. The resulting model, called ShapeContextNet, consists of a hierarchy with modules not relying on a fixed grid while still enjoying properties similar to those in convolutional neural networks --- being able to capture and propagate the object part information.

In addition, we find inspiration from self-attention based models to include a simple yet effective contextual modeling mechanism --- making the contextual region selection, the feature aggregation, and the feature transformation process fully automatic. ShapeContextNet is an end-to-end model that can be applied to the general point cloud classification. We observe competitive results on a number of benchmark datasets.

In this work, we investigate the structural information in typical problems in both

the machine learning and computer vision domains and propose effective yet efficient methods to tackle those problems. In particular, for structural labeling, we propose a fixed-point model which is able to learn/model long range structural contexts and is very efficient to train. For visual codebook learning, we propose a

randomness and sparsity induced learning scheme which can fit to the local intrinsic structure of image patches. We also propose methods to tackle the problems of mid-level feature learning and object tracking where the structural information are helpful. For mid-level feature learning, we propose a fully automatic algorithm which harvests effective visual concepts from a large number of Internet images using text-based queries. For object tracking, we propose a disagreement-based approach which can be built on top of nearly any existing tracking systems by exploiting their disagreements. Our methods achieve state-of-the-art performance and high efficiency.

Natura scenes consist of a wide variety of stochastic patterns. While many patterns are represented well by statistical models in two dimensional regions as most image segmentation work assume, some other patterns are fundamentally one dimensional and thus cause major problems in segmentation. We call the former region processes and the latter curve processes. In this paper, we propose a stochastic algorithm for parsing an image into a number of region and curve processes. The paper makes the following contributions to the literature. Firstly, it presents a generative rope model for the curve processes in the form of Hidden Markov Model (HMM). The hidden layer is a Markov chain with each element being an image vase selected from an over-complete basis, such as Difference of Gaussians (DOG) or Difference of Offset Gaussians (DOOG) at various scales and orientations. The rope model accounts for the geometric smoothness and photometric coherence of the curve processes. Secondly, it integrates both 2D region models, such as textures, splines etc with 1D curve models uner the Bayes framework. Because both region and curve models are generative, they compete to explain input images in a layered representation. Thirdl, it achieves global optimization by effective Markov chain Monte Carlo methods in the sense of maximizing a posterior probability. The Markov chain consists of reversivle jumps and diffusions driven by bottom up information. The algorithm is applied to real images with satisfactory results. We verify the results through random synthesis and compare them against segmentations with region processes only.

We present an algorithm for shape matching and recognition based on a generative model for how one shape can be generated by the other. This generative model allows for a class of transformations, such as affine and non-rigid transformations, and induces a similarity measure between shapes. The matching process if formulated in the EM algorithm. To have a fast algorithm and avoid local minima, we show how the EM algorithm can be approximated by using informative features, which have two key properties-invariant and representative. They are also similar to the proposal probabilities used in DDMCMC [13]. The formulation allows us to know when and why approximations can be made and justifies the use of bottom-up features, which are used in a wide range of vision problems. This integrates generative models and feature-based approaches within the EM framework and helps clarifying the relationship between different algorithms for this problem such as shape context [3] and softassign [5]. We test the algorithm on a variety of data sets including MPEG7 CE-Shape-1, Kimia silhouettes, and real images of street scenes. We demonstrate very effective performance and compare our results with existing algorithms. Finally, we briefly illustrate how our approach can be generalized to a wider range of problems including object detection.