Human beings have always been interested in understanding themselves and their surround- ings. Learning about the relationship between the two can reveal facts of the present and help predict the future, a critical part to live a better life. With the proliferation of IoT sensor devices, it is now possible to collect quality data for each individual and utilize this data for building personal models that can help to understand the self and environment. However, since this sensor data have different granularities and semantics, the semantic gap becomes even more formidable. Thus, there are challenges in aggregating, integrating, and synchronizing this heterogeneous data to a form such that it effectively describes the life experiences of each individual. In this dissertation, we design a personal chronicle, which contributes a solution to the aforementioned challenges, called Personicle, in which all kinds of personal data streams can be correlated with one another to form a model of a person.

To implement the Personicle, we first attempt to bridge the semantic gap between the low- level multimedia logs and high-level semantics by developing a common daily event model through the data unobtrusively obtained from smart devices. To do this, we define an atomic interval, which brings together the scattered sources of heterogeneous data to partition the data into manageable pieces. This atomic interval lets us segment a day into sequences of similar patterns and use the segments for daily event recognition.

Secondly, we design an event-triggered Ecological Momentary Assessment (EMA) to max- imize the chance of aggregating the semantic data from the users. Unlike the traditional EMA process, which mainly depends on user initiative and intervention, we contribute to overcoming the problems endemic to persistent data collection, such as missing a logging moment or early abandonment, by initiating the EMA process from the system side at the right moment.

Lastly, we propose a fully-automated approach to obtain latent semantic information from all the integrated data aiming to maximize the opportunity of both qualitatively and quan- titatively capturing one’s life experiences. To show a concrete example of this enrichment, we perform an experiment with “Eating” and “Working”, a complex event central to human experiences. These enhanced daily events can then be used to create a personal model that could capture how a person reacts to different stimuli under specific conditions.

With the proliferation of sensor data, a critical challenge is to interpret and extract knowledge from large-scale heterogeneous observational data. Most knowledge discovery frameworks relay on data mining techniques to extract interesting patterns. The problem of finding such patterns is NP-complete and the property of interestingness is not monotone since a pattern may be interesting, even if its subpatterns are not. In this dissertation a framework for interactive knowledge discovery from heterogeneous high-dimensional temporal data is presented. First, a high-level pattern formulation language is introduced. The language consists of an event model for fusing and abstracting data streams, a semi-interval time model for effectively representing temporal relations, and a set of expressive operators. Based on these operators, a visual and interactive framework is proposed which combines data-driven (bottom-up) and hypothesis-driven (top-down) analyses.

This framework takes advantage of data-driven operators for pattern mining and investi- gating unknown unknowns to generate a basic model and derive a preliminary knowledge. It also uses domain expert knowledge to guide the process of revealing known unknowns. An expert can seed a hypothesis, based on prior knowledge or the knowledge derived from data-driven analysis, and grow it interactively using hypothesis-driven operators. In the con- text of the pattern mining component, novel time efficient algorithms are introduced which allow discovery of hidden event co-occurrences from multiple event streams. A prototype of the framework is implemented as a web based system which can be utilized as an effective tool for explanation and decision making in almost all disciplines. The applicability of this framework is evaluated in a healthcare application for asthma risk management and a human behavior understanding application, called Objective Self. These applications and experiments highlight the actionable knowledge that the framework can help uncover.

Big data continues to grow and diversify at an increasing pace. To understand constantly evolving situations, data is collected from various location-based sensors as well as people using effective participatory sensing. Static sensors are placed at particular locations, monitoring and measuring important variables from the environment. Additionally, people contribute data in the form of mobile streams through participatory sensing. To process such disparate data for situation recognition, we need to address two major research challenges, namely data quality and data analysis.

This dissertation presents graph-based models as a general framework to combine diverse data sources for situation recognition. To improve spatial data quality, we propose a multimodal geospatial prediction model that integrates heterogeneous data into a unified space-time-value format, extracts multi-resolution spectral features, and predicts values at unobserved locations. We demonstrate the effectiveness of the proposed model using air pollution data. We then combine predicted air pollution data with pollen data to analyze the situation of asthma risk across California. The proposed model based on our predicted values provides greater accuracy in detecting high asthma risk areas compared to previous studies.

Given the wide use of mobile technology, humans also act as sensors and increasingly contribute photos as mobile data streams. We investigate the use of visual concepts for situation recognition and develop new models to tackle the technical challenges of noisy data and processing real-time data, which are problems in existing approaches. First, we propose a graph-regularized linear regression PCA to address photo clustering in a real-time setting. Second, to overcome the noisy data that degrades graph quality, leading to poor recognition results, we incorporate a capped norm into a graph embedding method to remove the adverse effects of outliers. Our models not only outperform existing approaches in overcoming the problems of noisy data and real-time data processing, but also allow us to utilize a new cluster of data through a soft label method. Finally, we apply the proposed approaches to Yahoo Flicker 100 Million photos for detecting evolving situations.

The real-time detection of personal life events has great potential to provide human-friendly services. Using detected life events in the fields of health care, smart homes, elderly surveillance, and smart car, etc. will provide more relational information and/or services to a user. However, automatically detecting a life event from human behaviors and their surrounding contexts is a challenging problem. I believe that the combination of Formal Concept Analysis (FCA) and a context-aware mobile computing system can help a path toward automated life event detection. The advancement of the smartphone and its embedded sensors will enable this scenario. I propose a generic context-aware system based on Formal Concept Analysis, which covers both front-end and back-end processing, to detect human life events. The main contributions of this system are: 1) it provides a framework for real-time personal monitoring; 2) it integrates, processes and stores personalized latent features; and 3) it combines heterogeneous data streams into one life event. The experimental validation, which I implemented on an Android platform and server, demonstrates that a general life event model can be applied to each individual and that concept data analysis can be substituted for statistical data analyses in life event detection. I believe that my findings can lead to new event detection approach, which is not just confined to specific environments, such as Social Life Networking, video streams, or artificially organized sentient locations, but can be opened for all environments in the real world by using sensors and context factors.

Visual information is now an increasingly important part of the data ecosystem. While photos are captured and shared widely over the internet, methods for image search and organization remain unintuitive. The current state of the art in image organization extracts people, objects, location and time information from images and provides options for the automatic grouping of photos based on one of these attributes. As a next step in image organization, we believe that images need to be grouped based on the events they represent. However, such an event-based grouping of images is presently very primitive and imprecise.

In this thesis, we present an event-based image organization approach, where the key-idea is to leverage visual concepts and spatio-temporal metadata of images in order to automatically infer their representative event; this approach combines clustering with probabilistic learning methods. Clustering is performed on images based on spatio-temporal metadata, where each cluster represents an event that occurred at a particular spatio-temporal point/region. We build probabilistic models that learn the associations between the different features and the predefined event labels of each cluster, and use the learned models to automatically infer the events in incoming photo streams. We evaluate efficiency of the proposed method by using a personal image data set, using metrics such as precision and recall.

The contributions of this thesis are two-fold. First, we use several web based sources for semantically augmenting the spatio-temporal metadata corresponding to the images, and second, we combine clustering with probabilistic learning to identify and annotate events in the photo stream, using the augmented metadata and visual image concepts.

The nature of data has changed from the past decades. A large volume of spatial-temporal data has been produced, collected, and shared to observe events in the physical world at different spatial scales and temporal frequencies. A multitude of data streams such as weather patterns, stock prices, social media, traffic information, and disease incidents can be used to recognize evolving real-world situations. These situations vary and affect multiple aspects of people’s lives - such as traffic, flash floods, economic recession, and epidemic diseases. Detecting situations in time to take appropriate actions can help in saving lives and resources. Building upon a situation recognition and social network concept, where people can easily connect to people online, the Social Life Network (SLN) concept is introduced. The goal is to connect people with the right resources efficiently, effectively, and promptly, depending on the evolving situations. To achieve this goal, a novel platform for situation recognition is required.

In this dissertation, we propose a generic framework for situation recognition from heterogeneous event streams. First, to ingest and store massive data streams, we implement the archival system using the pub/sub messaging system, Kafka, and big data management system, AsterixDB. Second, to handle data and events at different granularities, multi-resolution processing is introduced in every step including data ingestion, query processing, and data visualization. To aid users in finding the right resolution, statistical information about estimated error and response time for each situation model at different resolutions are provided. The system can automatically select an appropriate resolution or users can interactively select the most satisfactory resolution according to their needs. The idea that action drives design is a key component in creating a situation model. Finally, to fuse disparate data sources, a micro-reports concept is proposed. These micro-reports are compelling, universal, spontaneous, and objective. They can be used to solve problems of existing micro-blogs and replace them. The applicability of this framework is presented in a healthcare application for asthma risk management, disaster rescue for flood and hurricane mitigation, and smart city innovation for trash management in Downtown Washington D.C.

Population-based models are prevalent in different aspects of our lives. Decision-making in clinical health relies heavily upon the insights generated from population-based models such as recommended blood pressure and blood glucose values. These models and insights are easy to understand, but they do not necessarily capture the variance at an individual level. These are also likely to suffer from bias due to the participants' selection strategy (selection bias) or statistical limitations of the models used. Therefore, population-based models are not precise enough to make predictions at an individual level and highlight the need for personalized models for individual decision-making. N-of-1 analysis and modeling strategies are gaining popularity for addressing this problem. This modeling paradigm views every individual as a unique system and reduces variance at an individual level. Multimodal data generated by individuals and systems can be utilized to enable such analysis. Different wearable and IoT devices and smartphone applications capture data about individual lifestyles and health as daily life events and associated data streams. These event streams and data streams create a lifelog for the user that captures their habits and behaviors in the form of frequent event patterns.

In this work, we propose an interactive event analysis system that leverages the multimodal events and data streams from varied sources and enables analysts to perform N-of-1 analysis for individuals. The system is based on an event patterns language to represent temporal event relationships and extends the language to allow the creation of aggregate features from event patterns. The proposed system has three major components that allow the analysts to perform N-of-1 analysis:1. Event creation modules define new complex events from multimodal data using pre-defined event combination operators and motif discovery from data streams. Different event visualizations enable the analysts to explore the event space and identify event combinations for further analysis. 2. A hypothesis testing module allows the analyst to encode domain knowledge and personal beliefs in the form of a directed acyclic graph where every edge in the graph captures a causal relationship between the parameters (represented by the nodes). These relationships can then be tested using an implementation of the do-operator and estimate the effect of an intervention on the observed outcome. 3. A data-driven frequent event sequence detection module allows the analysts to discover frequent sequences of events and the time delays between the events leading up to an outcome event of interest. These frequent sequences describe the commonly observed events associated with the outcome and may represent a causal link between the events, which may be tested empirically or experimentally.

Life's most valuable asset is health. Continuously understanding the state of our health and modeling how it evolves is essential if we wish to improve it. Given the opportunity that people live with more data about their life today than any other time in history, the challenge rests in interweaving this data with the growing body of knowledge to compute and model the health state of an individual continually.

This dissertation presents an approach to build a personal model and dynamically estimate the health state of an individual by fusing multi-modal data and domain knowledge. The system is stitched together from four essential abstraction elements: 1. the events in our life, 2. the layers of our biological systems (from molecular to an organism), 3. the functional utilities that arise from biological underpinnings, and 4. how we interact with these utilities in the reality of daily life. Connecting these four elements via graph network blocks forms the backbone by which we instantiate a digital twin of an individual. Edges and nodes in this graph structure are then regularly updated with learning techniques as data is continuously digested. Experiments demonstrate the use of dense and heterogeneous real-world data from a variety of personal and environmental sensors to monitor individual cardiovascular health state.

State estimation and individual modeling is the fundamental basis for many revolutionary efforts in health. Continuous access to the individual's state allows a departure from disease-oriented approaches, to a total health continuum paradigm. Precision in predicting health requires a detailed understanding of an individual and their trajectory. By encasing this state estimation within a navigational approach, a user can use a systematic guidance framework to plan actions for each moment to transition their current state towards a desired one. This work concludes by presenting this framework of combining the health state and personal graph model to perpetually plan and assist us in living life towards our goals.

Personalized Food Recommendation Systems (Food-RecSys) are recognized for their potential to enhance dietary practices and address pressing health issues. Despite their promise, current food recommendation methodologies, predominantly rule-based and classification-driven, encounter substantial barriers in achieving practical applicability. This challenge is largely attributed to two main factors: the poor understanding of food recommendation specific components as a whole and inadequate research on a holistic integration between all the key components which enables the final methodology to achieve effective results, and also, the limitations of conventional machine learning models, which falter in the face of a virtually limitless array of food categories and the inherent imbalance within datasets. In this context, the advent of Large Language Models (LLMs) presents an intriguing alternative, suggesting a path forward for recommendation systems. Nevertheless, existing approaches that employ LLMs generally adopt a generic Recommendation as Language Processing (RLP) strategy, which falls short of incorporating the nuanced components essential for successful Food-RecSys.

To bridge this research gap, this thesis provides a comprehensive overview on an intensely complex big picture, identifies the key components and provides innovative exploratory models for each the multiplex key components required for personalized contextual food recommendation, additionally, articulates a novel feasibility study of a comprehensive integrated personal contextual Food-RecSys framework, underpinned by several pivotal innovations aimed at fortifying the effectiveness of personalized food recommendations. Key among these is the development of a multimedia food logging platform and the World Food Atlas, the latter of which facilitates geolocation-based food queries—a capability notably absent in current offerings. Moreover, this work pioneers the Food Recommendation as Language Processing (F-RLP) framework. F-RLP represents a bespoke solution that adeptly harnesses the strengths of LLMs, furnishing a food-specific recommendation infrastructure that transcends the limitations of generic models.