Describing visual image contents by semantic concepts is an effective and straightforward way to facilitate various high level applications. Inferring semantic concepts from low-level pictorial feature analysis is challenging due to the semantic gap problem, while manually labeling concepts is unwise because of a large number of images in both online and offline collections. In this paper, we present a novel approach to automatically generate intermediate image descriptors by exploiting concept co-occurrence patterns in the pre-labeled training set that renders it possible to depict complex scene images semantically. Our work is motivated by the fact that multiple concepts that frequently co-occur across images form patterns which could provide contextual cues for individual concept inference. We discover the co-occurrence patterns as hierarchical communities by graph modularity maximization in a network with nodes and edges representing concepts and co-occurrence relationships separately. A random walk process working on the inferred concept probabilities with the discovered co-occurrence patterns is applied to acquire the refined high level image semantic representation. Through experiments in applications including automatic image annotation, semantic image retrieval, moth species identification and multi-pedestrian tracking on several challenging datasets, we demonstrate the effectiveness of the proposed concept co-occurrence patterns as well as the proposed image semantic representation in comparison with state-of-the-art approaches.

Facial expression plays a significant role in human communication. It is considered the single most important cue in the psychology of emotion. Facial expression is taken as a universally understood signal, which triggers a discrete categorical basic emotion, including joy, sadness, fear, surprise, anger, and disgust. Thus, automatic analysis of emotion from images of human facial expression has been an interesting and challenging problem for the past 30 years. Aiming towards the applications of human behavior analysis, human-human interaction and human-computer interaction, this topic has recently drawn even more attention.

Automatic analysis of facial expression in a realistic scenario is a much more difficult problem due to that the 2-D imagery of human facial expression consists of rigid head motion and non-rigid muscle motion. We are tasked to solve this "coupled-motion" problem and analyze facial expression in a meaningful manner. We first proposed an image-based representation, Emotion Avatar Image, to help person-independent expression recognition. Second, an real-time registration technique is designed to improve frame-based streaming action unit (AU) recognition. The proposed accurate expression recognition techniques are then applied to the field of advertising, where audiences' commercial watching behavior is thoroughly analyzed.

Robots are utilized in activities such as mine detection, search and rescue, etc. Especially where the environment is dangerous or search time can be optimized, etc. As the cost of robots reduced and application of robotic systems grew, researchers started working on distributed robotic systems. However, creating a robust, flexible and scalable system of collaborating mobile robots that can efficiently perform the search task has been a challenging issue for roboticists. Biologists observed that animal interactions in their societies create a collective behavior that helps them in foraging, group defending and other activities. They show that each animal in its society has a simple role but the collection of these behaviors enables them to accomplish a complex task. Moreover, swarm behavior has interesting properties such as robustness, flexibility and scalability. Based on these studies, researchers have introduced new optimization methods. Roboticists utilize these optimization methods to provide important properties of swarm behavior. PSO is one of the successful optimization methods in this area. Initially, PSO performs well in exploring different search regions, however, in some cases the method doesn't exploit well in promising regions which increases the search time. In this study, mobile robots are deployed to find a target in the search space. Robots interact with each other and move around the search space using PSO algorithm. In addition, the MPSO algorithm is introduced to create an efficient balance between exploration and exploitation of the PSO algorithm. Results show that MPSO reduces search time and minimizes region revisiting.

Vehicle classification has become a very important subject of study because of its importance in autonomous navigation, surveillance and traffic analysis. Classification of vehicles from the rear view is challenging because all the vehicles have subtle appearance differences from the rear view, changing illumination conditions, presence of shadows and real-time considerations. While numerous approaches have been introduced for this purpose, no specific study has been conducted to provide a robust and complete video-based vehicle classification system based on the rear view where the camera’s field of view is directly behind the vehicle. In this paper we present a multi-class vehicle classification system which classifies a vehicle into one of four possible classes Sedan, Minivan, SUV and Pickup truck when seen from its rear view. For a given geometric setup of the camera we use a feature set of Visual Rear Ground Clearance, height of the vehicle and perpendicular distance between the bottom of the license plate and bottom of the rear bumper for classifying the vehicle. Results are shown on large data-sets of freeway videos.

Recent advances in unmanned aerial vehicles and camera technology have proven useful for the detection of smoke that emerges above the trees during a forest fire. Automatic detection of smoke in videos is of great interest amongst the Fire Department. To date, in most parts of the world, the fire is not detected in its early stage and turns catastrophic. This thesis introduces a novel technique that integrates temporal and spatial features in a deep learning framework using semi-supervised spatio-temporal video object segmentation and dense optical flow. However, detecting this smoke in the presence of haze and without the labeled data is difficult. Considering the visibility of haze in the sky, a dark channel pre-processing method is used that reduces the amount of haze in frames and consequently improves the results. Online training is performed on a video at the time of testing that reduces the need for ground-truth data. Tests using public datasets and videos show that the proposed algorithms outperform previous work and are robust across different wildfire-threatened locations.

Person Identification or recognition has been receiving broad interests and it is highly desirable in applications such as security monitoring, authentication, etc. In order to recognize a person, different traits, including fingerprint, face, and gait, can be used. Among these possible traits, face and body are preferred since they can be acquired without the person's cooperation. In controlled environment, recognition is less challenging with well posed subject in high resolution. However, in real-world scenarios, where the image of a person exhibits variations in pose, illumination, and resolution, standard pattern recognition methods may fail. Driven by the necessity for person identification in real-world, we have proposed several identification methods. Specifically, we have developed a face image super-resolution method as a pre-processing step to improve the face recognition accuracy. In addition, to recognize person in a surveillance setting with multiple cameras, we have developed an algorithm that utilizes multiple cameras for face recognition by encoding the person-specific dynamics with a dynamic Bayesian network. In case the face of a person cannot be reliably acquired, identifying person by body appearance is preferred. To this end, we have proposed two methods to identify person in multiple surveillance cameras, using a novel reference descriptor and a sparse representation, respectively. To validate the proposed method in this dissertation, we have conducted extensive results on publicly available datasets. Results show that each of the aforementioned method achieves state-of-the-art performance in various person identification tasks.

In the dissertation, two advanced computer vision techniques, named symmetry and feature selection, are proposed. The wide existence of symmetry in many image objects generates the motivation of using symmetry as a high level feature in region growing image segmentation and region-of-interest (ROI) detection in brain MRI sequences. The symmetry is explicitly applied in different forms as symmetry affinity matrix, high-level segmentation cue, statistical analysis and 3D asymmetry volume in classification features. The incorporation of symmetry provides a new effective feature to achieve the performance improvement. In the second field of my research, the feature selection with Sequential Floating Forward Selection (SFFS) as the search strategy, and with the Bayesian classifier as the evaluation metric, is applied in content-based image retrieval (CBIR), semi-supervised learning with relevance feedback, local kernel based distance metric, image classification, and online ensemble learning. It provides more compact and optimal feature sets to generate robust learning models. Experimental results on wide range of image datasets indicate the advantages of using symmetry and feature selection in computer vision tasks.