Internet-of-Things (IoTs) have developed for more than one decade since their first appearance. At their early stage, IoT devices are generally developed as sensor extensions, which are not capable of performing complicated computing tasks. Hence, offloading the computational tasks to the cloud servers is the only option to learn the information from the sensory data. However, with the rapid evolution of the technologies in the field of System-on-Chip (SoC), more complex computational tasks are now able to be performed on IoT devices, which is inevitably facing multi-tenancy requirements. For example, a UAV-as-a-Service can be used by different users to perform object detection and traffic monitoring tasks. Unfortunately, current multi-tenancy edge systems face several issues from the security of the sensors and actuators and the safety of their cyber-physical environment.

In this dissertation, we build an edge computing framework to address the security and safety problems of current multi-tenancy edge systems. This framework focuses on a broad category of IoT devices with sensors and actuators, including both rich computing resource devices with powerful edge processors such as ARM Cortex-A series processors and bare-metal IoT devices with only microcontrollers such as ARM Cortex-M series processors. The first part of this work focuses on securing the sensors and the actuators at the edge. More specifically, we propose PROTC and VirtSense that utilize the hardware-assisted trusted execution environment (TEE) such as ARM TrustZone to isolated the actuation and sensing service from the untrusted execution environment. Furthermore, for those bare-metal IoT devices that the hardware-assisted TEE is not available, we design Aerogel that utilizes WebAssembly (Wasm) initially designed for secure JavaScript execution environment in web browsers, to protect the sensors and actuators. Our results show that the protection of Aerogel introduces 0.14% to 1.04% overhead to our benchmark tasks.

After the sensors have been adequately secured, the second part of this dissertation focuses on the data usage from the sensors by preventing the applications from extracting unexpected information from the sensory data, which could ultimately lead to the leakage of users' privacy. To achieve such a goal, we design a performant and secure runtime named SecDeep that protects the inference libraries' integrity and sensor data's confidentiality by leveraging the TEE to securely host the deep learning code and the drivers of sensors, actuators, and the edge accelerator. Our results indicate that with the assistance of the edge accelerator, SecDeep achieves 16 to 172 times faster performance than without using it while still providing data protection.

Although ensuring the security of the IoT devices can mitigate the safety issues, not all safety issues are caused by security problems such as the user's misconfiguration. We propose RemedIoT in the third part of this dissertation, which tackles the IoT device actuation conflicts by introducing actuation programming abstractions to provide remedial actions of those conflicts. Our evaluations on RemedIoT shows that around 80% of device conflicts can be provided remedial actions.

Recent changes in pervasive sensing applications require software systems that can address diverse technical, architectural, and human issues. Research on wireless sensor networks has served as technical fundamentals, mobile sensing research has helped solving many architectural problems, and now various human/cultural difficulties in pervasive sensing systems are revealed. We compare two contrasting architectural styles, i.e., the cathedral and the bazaar, and discuss the design of a system that unifies the both. Main challenges in designing such a system include: (1) a large amount of personal data; (2) privacy in sharing them; (3) energy-efficiency on mobile devices. We address them using a distributed network of virtually-private data stores featuring rule-based sharing control and flow-based execution of context inferences. Our performance benchmarks show that the rule processing delay is less than 25 ms in typical usage scenarios, and the flow-based execution saves 38.3% of CPU time as well as 54.3% of memory usage in comparison to a bus-based framework. Our twelve-person user study results indicate participants feel less privacy concerns using the rule-based sharing control. We also discuss an interesting tradeoff between usability and controllability, discovered from the user study. Finally, all source code for this research is readily available online.

Affordable, wearable, embedded, wireless medical sensor systems that

enable continuous long term monitoring of physiological signals could

revolutionize health care. Realizing this vision demands devices that

are small, unobtrusive and low power. Effectively inferring health

conditions begins by acquiring physiological signals of interest and

decisions made about what signals are acquired, when, where and at

what rate affect not only the energy efficiency of the sampling

process but also that of other downstream components in the signal

processing chain.

While the Nyquist sampling theorem provides for

exact reconstruction from discrete-time samples,

the prescribed rate is often wasteful for physiological sensing applications

since it neither exploits the structure of signals fully nor does it take into

account that many applications don't require full reconstruction at all.

This dissertation illustrates how energy efficiency of the entire system can be improved

by targeting just the signal acquisition process while being cognizant of the entire

sensing information stack, from sampling, processing and communication to the

top-level application inferences.

A key ingredient that makes optimizing the sensing stack worthwhile is that

the sampling stage, which is usually abstracted away from the system,

can now utilize sophisticated methods that have emerged in the past few years.

Recent advances in sampling and recovery techniques have demonstrated

considerable rate reductions by employing stronger models of the phenomenon

coupled with application-specific objectives (detection or control vs.

reconstruction), which potentially translates to higher energy, processing and

communications efficiency at the system level.

This research describes four major thrusts that span the processing chain from

hardware to algorithms to inferences. First, recognizing that signal conditioning

front-end circuits could account for a large portion of the energy expenditure

in low power sensing, we demonstrate how prudently duty cycling them could

increase device lifetime by threefold and reduce data rate by almost fourfold

for an electrocardiography monitor.

Then, we go on to show how one could further

slash data rates using the new theory of compressed sensing. For a neural spike recorder, we exploit the fact that action potentials have both a structure and short term stability in their morphology. This meant that we could utilize historical signal information to optimize and adapt compressed sensing recovery, with only receiver-side modifications, doubling the compression ratio.

Third, since body area networks are prone to congestion and interference,

we propose a rate control algorithm for the wireless channel so that the

most important data from the most informative sensors gets delivered

for maximum inference quality. Finally, we prove that compressed

sensing could be utilized not only to compress signals but could also improve the robustness of sensor transmissions at low computational cost by viewing it as joint source-channel coding for wireless erasure channels.

Electrolocation is a method of sensing found in weakly electric fish that utilizes electrical discharges to sense objects and navigate through their environment. Inspired by these biological findings, this thesis will describe a modeling and processing method to emulate an application using this sensing modality. We approached the problem with System Identification to estimate a non-parametric model instead of relying on complex physical equations. We will describe how a Kalman filter and an estimation function uses the model to process incoming sensory information. From our analysis, we show improvements using online adaptive model over offline training. Finally we will compare experimental results estimating object distance using linear regression and SVM.

Real-world scenes are typically complex and dynamic. Intelligent systems must consistently analyze their surroundings using sensory data to develop situational awareness. Scene analysis based on static images and videos has achieved great success in tasks like object detection and classification. Research efforts have also been made around wireless sensing to acquire more knowledge of the scene, such as recognizing human activities. However, there is still a substantial amount of information from the object that is difficult for standard cameras or existing RF sensing systems to perceive. Meanwhile, while multimodal sensors have been applied to scene analysis, such systems often focus on achieving better performance on a specific sensing task without combining the complementary capabilities of these sensors to fully leverage their strengths. Furthermore, although various deep learning architectures have been proposed for multimodal sensor data processing, they often experience robustness issues, such as degraded performance in adverse sensing conditions. To address these challenges, we investigate the sensing capabilities of radio-frequency (RF) sensors and their fusion with other sensing modalities, such as camera and depth sensors, for deeper scene analysis. Here, our concept of "deep" includes both the richness and robustness of inferences about the scene. Our key insights can be summarized in three parts. First, RF sensors reveal physical phenomenons about objects that are otherwise concealed, such as vibrations.

The dissertation introduces an ultra-wideband (UWB) radar-based wireless vibrometry sensing system, UWHear, which is capable of accurately recovering and separating vibrations from multiple sources and is resilient to non-target noise. Second, the fusion of wireless vibrometry with video scene analysis forms a more comprehensive understanding of the scene. We propose a real-time RF-vision sensor fusion system, Capricorn, that efficiently builds a cross-modal correspondence between visual pixels and RF time series. Capricorn leverages the complimentary sensing capabilities of the RF and vision sensors. It not only captures the extrinsic properties of objects in a scene, such as their shape, type, and location, but also reveals the objects' latent properties, including vibrations and vital signals, providing insights into the objects' internal physical or biological states. Third, we can make a scene analysis system resilient to adverse sensing conditions using a thoughtfully designed deep-learning-based multimodal sensor fusion architecture. We created a multi-node multimodal sensor network system, collected a 9-hour dataset (GDTM) for multimodal object localization, and designed neural network architectures (FlexLoc) for robust multimodal sensor information fusion. Specifically, in FlexLoc, we investigated how to use conditional neural networks to make the deep learning architecture robust to sensor perspective shifts. Through this dissertation, we contribute to the construction of a deeper environment perception with a combination of signal-processing algorithms, deep-learning strategies, and system design.

Social networks like Facebook and MySpace are on the rise and gain more and more users every day. They link together people and their frinds for messaging, organizing parties and events, or to keep up to date with whome is doing what where. SenseShare is an application that exploits these social networks and structures to provide authentication, privacy, and security while sharing sensor data with other people.

The number of audio-sensing-related applications is growing rapidly, such as the voice assistant as an interface between humans and computers, and the automatic-speaker verification system, which involves personal identity. These applications demand reliability and security of the audio sensing system. For example, an audio recognition system can easily get confused by the sound of non-target objects, as everything is fused in the collected audio. Meanwhile, a speaker verification system may fail under spoofing attacks of the computer-generated audio.

In this work, we focus on reinforcing existing audio sensing technologies to make it more robust and secure. This work comes in two parts. In the first part, we explore how we can leverage other modalities to improve the reliability of audio sensing, such as the impulse-radio Ultra-wideband (IR-UWB) radar. Our experiments show that this IR-UWB audio-sensing system can penetrate light-building materials to recover the sound. Meanwhile, the system is capable of measuring the distance between the sound source and the sensor, with which we can easily recover and separate the sound from multiple sources. In the second part, we explore how to defend against state-of-the-art acoustic attacks for critical applications such as voice authentication. We build a deep-learning-based system designed to determine if an audio clip is genuine human speech or, on the other hand, a computer-generated or a replayed one. This system is designed to work along with the automatic speaker verification system to protect it from spoofing attacks. Our results show a significant improvement from the baseline and some generalization abilities on unseen attack types. The work presented in this thesis provides the preliminary steps towards utilizing multiple modalities for robust audio sensing applications across a variety of environments, as well as an extra anti-spoofing protection for these applications using deep learning.

Over the past decade, deep learning models have gained significant popularity for processing noisy sensor inputs in various real-world scenarios. However, they still face challenges such as data inefficiency, interpretability limitations, and weak reasoning abilities. On the other hand, Symbolic algorithms offer high-level reasoning and interpretability but struggle with uncertainties and high-dimensional sensor data. Neuro-symbolic approaches have emerged as promising solution, combining the strengths of deep learning and symbolic methods. They show potential for Complex Event Processing (CEP) in critical domains like healthcare monitoring and smart city control. This work explores the performance of neuro-symbolic methods in complex event detection tasks from multimodal sensor data, which involves recognizing complex events with complicated temporal patterns that span a wide time range. However, the lack of multimodal and complex event datasets poses a challenge. To address this, we formulate the complex event detection problem, synthesize a multimodal activity dataset, and build a stochastic daily human activity simulator to generate multimodal complex event datasets. We design a two-module system that includes a multimodal fusion module to handle sample inconsistencies and utilize information across modalities, and a backbone module that can be replaced with neural-only and neuro-symbolic models for performance comparison. The evaluation results demonstrate the superiority of the neuro-symbolic approach in complex event detection and provide insights for developing improved neuro-symbolic architectures for effective and applicable complex event detection systems in multimodal settings.

There has been a significant amount of recent work towards fooling deep-learning-based classifiers, particularly for images, via adversarial inputs that are perceptually similar to benign examples. However, researchers typically use minimization of the $L_p$-norm as a proxy for imperceptibility, an approach that oversimplifies the complexity of real-world images and human visual perception. We exploit the relationship between image features and human perception to propose a \textit{Perceptual Loss (PL)} metric to better capture human imperceptibly during the generation of adversarial images. By focusing on human perceptible distortion of image features, the metric yields better visual quality adversarial images as our experiments validate. Our results also demonstrate the effectiveness and efficiency of our algorithm.