The rise of autonomous and artificially intelligent systems promises to deliver efficient and optimal performance as well as unprecedented functionality in a variety of fields and human endeavors. As varied as these fields are, a common property of most autonomous systems is that they convert raw data about their relationship with the environment into optimal actions to achieve a desired goal. One may further divide this conversion into the sub-tasks of sensing and perception, planning, and actuation. Perception is an important part of this pipeline since it describes the set of algorithms for turning raw data into useful bits of information that can be used for planning.

At its core, we may view perception using an estimation framework. Perception algorithms essentially seek to estimate information from noisy, incomplete and raw data obtained from sensors. However, the accuracy of the information obtained (and consequently effectiveness of the actions taken by the system) are dependent on the assumptions and models used in designing perception algorithms. Therefore, it is important to take care in designing accurate but tractable models for perception. Unfortunately, many perception tasks involve complex functions and sensor models relating observed data to information. These functions are difficult to model from physical properties. In these cases data-driven methods can provide complementary techniques that result in excellent models for very complex systems.

In this dissertation, we present three perception algorithms designed for applications in Autonomous Driving, Energy Systems and Mixed Reality. We make use of data driven methods to provide approximations to complex sensor models and present tractable estimation algorithms for turning raw data into information. We verify our approach on both synthetic and real data and report excellent results.

Supervisory Control and Data Acquisition (SCADA) systems are deeply ingrained in the fabric of critical infrastructure sectors. These computerized real-time process control systems, over geographically dispersed continuous distribution operations, are increasingly subject to serious damage and disruption by cyber means due to their standardization and connectivity to other networks. However, SCADA systems generally have little protection from the escalating cyber threats. To achieve defense-in-depth for SCADA systems by means of intrusion detection and resilient control, this dissertation strives for a robust stochastic signal and system approach without being overly-pessimistic. Its main elements are (1) two SCADA-specific comprehensive taxonomies with one on cyber attacks and the other on intrusion detection system to layout the lay of the land and shed light to the workspace, (2) one overall framework/architecture for intrusion detection and resilient control -- Xware (3) its measurement fusion assurance component -- Trust counter, (4) one signal-based early-detection and resilient estimation scheme with proved theoretical performance bounds, for SCADA systems in general. Especially the said Robust General Likelihood Ratio Test (RGLRT) is generic enough and has been applied to linear dynamical systems in general and beyond. (5) The application of RGLRT in network traffic anomaly detection. (6) The application of RGLRT to anomaly detection for SCADA systems in smart grids through model construction and identification for both clean renewable energy supply and variable consumer demand.

First, in order to understand the potential danger and to protect SCADA systems, we highlight their difference from standard Information Technology (IT) systems are and present a set of security property goals. Furthermore, we systematically identify and classify likelycyber attacks including cyber-induced cyber-physical attacks on SCADA systems according the SCADA's hierarchy. Determined by the impact on control performance of SCADA systems, we use the attack categorization criteria to stress commonalities and important features of such attacks that define unique challenges posed to securing SCADA systems versus traditional IT systems.

Second, in order to address the big challenge of how to modify conventional IT intrusion detection techniques to suit the needs of SCADA, we explain the nuance associated with the task of SCADA-specific intrusion detection and frame it in the domain interest of control's researchers to illuminate problem space. We present a taxonomy and a set of metrics for SCADA-specific intrusion detection techniques through heightening their possible use in SCADA systems. In particular, we enumerate a list of Intrusion Detection Systems (IDS) that are proposed to undertake this endeavor. Drawing upon the discussion, we identify the deficits and voids in current research. Based upon this taxonomy and analysis on which SCADA-specific IDS strategies are most likely to succeed, we offer recommendations and future research venues in part through presenting a prototype of such efforts towards this goal.

Third, we present the overall architecture for instruction detection and resilient control Xware. It is comprised of two strong footings -- Normalcy Checking, a control theoretic, domain knowledge specific, specification-based payload inspection system and

a high-speed, real-time, behavioral-based Network Intrusion Detection System (NIDS). Xware integrates a Trust Counter to verify the truthfulness of sensor measurements. It also provides exfiltration of confidential information from within the intranet. Moreover, Xware hardens SCADA system with compensation schemes when intrusion evades NIDS

or unexpected fault occurs to guarantee its performance. It puts things in perceptive and highlights the overall systematic and holistic approach.

Fourth, we propose the Trust Counter to deal the case when the possible manifestation of those potential disruption from cyber attacks can affect the Kalman filter, the primary recursive estimation method used in the control engineering field. Whereas, to improve such estimation, data fusion may take place at a central location to fuse and process multiple sensor measurements delivered over the network. In an uncertain networked control system where the nodes and links are subject to attacks, false or compromised or missing individual readings

can produce skewed results. To assure the validity of data fusion,

we propose a centralized trust rating system. It evaluates the trustworthiness of each sensor reading on top of the fusion mechanism. The ratings are represented by Beta distribution, the conjugate prior of the binomial distribution and its posterior. Then an illustrative example demonstrates its efficiency.

Fifth, RGLRT is an earlier anomaly detection and resilient estimation scheme for the cyber-physical systems, networked control systems to be specific, in an uncertain network environment. It robustly identifies and detects outliers among real-time multidimensional measurements of dynamical systems by using an online window-limited sequential Robust Generalized Likelihood Ratio (RGLR) test without any prior knowledge of the occurrence time and distribution of the outliers. The robust sequential testing and quick detection scheme achieves the optimal stopping time with low rates in both false alarm and misdetection. We propose a set of qualitative and quantitative metric to measure its optimality in the context of cyber-physical systems. Further, this resilient and flexible estimation scheme robustly rectifies and cleans data upon both isolated and patchy outliers while maintain the optimality of the Kalman Filter under the nominal condition. Its approximated optimality of the robustification performance is shown through stochastic approximation.

Sixth, we give a network anomaly detection scheme as one of the applications of RGLRT. The time series model of Autoregressive Integrated Moving Average (ARIMA) progress, finds its wide usage including network security applications. Model building and anomaly detection based on such models are often a first and important step towards monitoring unexpected problems and assuring the soundness and security of those systems being studied. The time variability by the coefficients in those dynamic regression models is particularly relevant and possibly indicative. To address this issue, a corresponding framework and a novel anomaly detection approach based on the Kalman filter for identifying those dynamic models including their parameters and a General Likelihood Ratio (GLR) test for detecting suspicious changes in the parameters and therefore the models is proposed. The idea is shown through experiments and show its promising potential in terms of accuracy and robustness.

Seventh,we apply RGLRT to anomaly detection for SCADA systems in smart grids. While the utilization of clean energy resources including wind and solar power sets to grow from filling the gap of peak hours to taking a larger share in the upcoming smart grid and efficient infrastructure, the price-incentivized electricity consumption shall alleviate peak hours and reduce power outages. Both benign faults and malicious attacks threat the reliability and availability of the new grid. We address these duo problems from the angle of one fundamental technique used. The ARIMA time series models play roles at both ends in this new ecosystem: namely, predicting the variable clean energy resource on the supply side and forecasting the flexible load demand on the consume side. Model construction and anomaly detection based on such models are often a first and important step towards monitoring unexpected problems and assuring the soundness and security of those systems being studied. The time variability of the coefficients in those dynamic regression models is particularly relevant and possibly indicative. Thus a corresponding framework and a novel anomaly detection approach is introduced. It's based on a robustified Kalman Filter for identifying those dynamic models including their parameters and a RGLRT for detecting suspicious changes in the parameters and therefore the models. Currently, the effectiveness and robustness of this method is shown through simulation.

Technological advances in communication and computing coupled with low costs of sensors have enabled large-scale deployment of video camera networks in our environment nowadays. They are increasingly being used by decision-makers for perceiving elements of an environment in real-time, comprehending their meaning and predicting their status in the near future. Such situational awareness is crucial in dynamic scenarios that arise in military command and control, facility security and emergency services such as policing and fire-fighting. While human perception driven situation awareness has worked well in constrained settings, it is unfortunately not scalable. Furthermore, scenarios such as security and surveillance commonly involve highly complex cognitive tasks that can quickly become monotonous and mentally taxing for human operators. In this thesis, we present new frameworks for automating the perception stage of situation awareness.

We begin this thesis with the development of a system that is capable of categorizing objects and landmarks in an efficient and distributed manner. Our system is designed to operate with net- works of wireless smart cameras for local perception, and a central station for global inference. We demonstrate that this decoupling of the algorithm pipeline can drastically minimize the power and bandwidth consumed by the wireless cameras. Further, we experimentally validate that our multiple-view inference framework can significantly improve the performance of object and land- mark categorization over traditional single-view settings.

In the second part of the thesis we extend our distributed object categorization framework to address the problem of automatic human activity detection and categorization. We are particularly interested in the development of rich representations for human motion that are invariant to perspective, scale and the speed at which actions are performed. We propose a generalized framework to perform spatiotemporal fusion of dynamic imagery from multiple wireless smart cameras, and validate the efficacy of our fusion framework on both publicly available, and novel datasets.

In most realistic scenarios that require situation awareness, objects and people occur in cluttered scenes and exhibit immense variability in their appearance, shape and pose. In the final part of this thesis, we analyze the interplay between computer vision tasks such as segmentation and categorization and present joint frameworks that significantly improve the performance of each task. Our experimental analysis demonstrates that detection and categorization hypotheses help provide good segmentation results, and that segmentation can be used to prune errors in the hypothesis.