UC San Diego

This dissertation focuses on vision-based articulated body pose tracking and human activity analysis for interactive applications, e.g., intelligent driver assistance systems, gesture-based interactive games, and smart rooms. Although there has been a considerable amount of related research effort, developing real-time, robust, and efficient vision -based systems for real-world interactive applications is still an open and important research area. Since human activities in the real-world may happen at different levels of detail (e.g., full body, upper body, hands, head, and feet), it is desirable to have systems that look at humans at multiple levels for better understanding of their activities. In addition to common computer vision issues such as occlusion, background clutter, variable lighting condition, and human appearance, there are other research issues that need to be addressed, with the objective of analyzing human posture at multiple levels for interactivity. This dissertation discusses those issues and proposes several relevant frameworks and approaches for human posture and activity analysis at different levels of detail. First, in order to achieve real-time performance and robustness required for interactive applications, the trade-offs in developing generic versus application-specific approaches for efficiency should be considered. Focusing on applications like driver assistance systems and smart meeting rooms, we develop the very first system, as far as we are concerned, that does both real-time upper body pose tracking in 3-D by observing extremity movements and then gesture recognition using pose tracking output. Second, it is more feasible to a deploy multilevel posture analysis system if we have efficient algorithms which may apply to different body levels. In that regard, we develop an integrated framework with automatic initialization for body and hand modeling and tracking from 3-D voxel data. We also develop an optical flow-based framework for driver foot and head behavior modeling and prediction which shows potential in mitigating the incidents of pedal misapplication in the real world. Third, we develop a driver assistance system that combines information from driver head and hand activities for distraction monitoring. Lastly, we present our development of multimodal driving experiments and analysis based on the ability to track driver activities at different levels which provides insight into the effect of audio and visual cues on driver behavior