This dissertation focuses on human behavior modeling and incentive designs for online platforms, specifically in two application domains: collective intelligence platforms and healthcare intervention platforms. In the last two decades, wide adoption of internet facilitates the emergence of online collective intelligence platforms. In the mean time, the fast developing mobile technology enables real-time healthcare interventions. This dissertation is a step towards implementing human behavior modeling through machine learning and optimization techniques to enhance the efficacy of these online platforms. Case studies show that human behavior modeling combined with smart incentive designs have the potential to improve the performance of such online platforms.

In the first part of this dissertation, I present two collective intelligence platforms: M-CAFE and DebateCAFE. M-CAFE is a mobile-friendly platform that encourages students to check in weekly to numerically assess their course performance, provide textual ideas about how the course might be improved, and rate ideas suggested by other students. For instructors, M-CAFE displays ongoing trends and highlights potentially valuable ideas based on collaborative filtering. M-CAFE is complementary to existing platforms, such as Piazza and stackExchange and allows students to step back and consider their own performance and the performance of their instructors, filling the gap between voluminous transcripts from existing platforms and a one-time-only, end-of-course evaluation. DebateCAFE is an online deliberation platform that introduces a novel incentive mechanism to encourage participants to articulate persuasive arguments on both sides of a complex issue. It uses a combination of uncertainty sampling and collaborative filtering to mitigate bias from selective exposure and highlight/rank the persuasive arguments. Furthermore, DebateCAFE assigns a score to each participant based on the lower of the Wilson scores of the two arguments entered to encourage strong arguments for opposing opinions. Both platforms were built upon the CAFE framework, which is based on Opinion Space.

In the second part of this dissertation, I describe a novel mobile phone application (app) called CalFit and introduce the Discontinuation Prediction Score (DiPS) for non-adherence prediction. CalFit implements important behavior-change features like dynamic goal setting and self-monitoring. Specifically, CalFit uses a reinforcement learning algorithm to generate personalized daily step goals that are challenging but attainable. Two empirical studies with university staff and students indicate that dynamic goal setting is effective in promoting physical activity. Despite the success, I identify that low adherence is a major drawback for mobile-based interventions like ours. Therefore, I further developed the Discontinuation Prediction Score (DiPS), which uses objectively measured past data (e.g., steps and goal achievement) to provide a numerical quantity indicating the likelihood of exercise relapse for the upcoming week for each subject. I present two versions of DiPS using logistic regression and support vector machine methods to demonstrate that DiPS has potential to accurately predict exercise relapse and efficiently allocate resources to improve compliance.

The internet enables us to collect and store unprecedented amounts of data. We need better models for processing, analyzing, and making conclusions from the data. In this work, crowdsourcing is presented as a viable option for collecting data, extracting patterns and insights from big data. Humans in collaboration, when provided with appropriate tools, can collectively see patterns, extract insights and draw conclusions from data. We study different models and algorithms for crowdsourcing discovery.

In each section in this dissertation a problem is proposed, the importance of it is discussed, solutions are proposed and evaluated. Crowdsourcing is the unifying theme for the projects that are presented in this dissertation. In the first half of the dissertation we study different aspects of crowdsourcing like pricing, completion times, incentives, and consistency with in-lab and controlled experiments. In the second half of the dissertation we focus on Opinion Space\footnote{opinion.berkeley.edu} and the algorithms and models that we designed for collecting innovative ideas from participants. This dissertation specifically studies how to use crowdsourcing to discover patterns and innovative ideas.

We start by looking at the CONE Welder project\footnote{Available at http://cone.berkeley.edu/ from 2008 to 2011} which uses a robotic camera in a remote location to study the effect of climate change on the migration of birds. In CONE, an amateur birdwatcher can operate a robotic camera at a remote location from within her web browser. She can take photos of different bird species and classify different birds using the user interface in CONE. This allowed us to compare the species presented in the area from 2008 to 2011 with the species presented in the area that are reported by Blacklock in 1984 \cite{Blacklock:1984}. Citizen scientists found eight avian species previously unknown to have breeding populations within the region. CONE is an example of using crowdsourcing for discovering new migration patterns.

Crowdsourcing can also be used to collect data on human motor movement. Fitts' law is a classical model to predict the average movement time for a human motor motion. It has been traditionally used in the field of human-computer interaction (HCI) as a model that explains the movement time from an origin to a target by a pointing device and it is a logarithmic function of the width of the target ($W$) and the distance of the pointer to the target ($A$). In the next project we first present the square-root variant of the Fitts' law similar to Meyer et el. \cite{meyer1988optimality}. To evaluate this model we performed two sets of studies, one uncontrolled and crowdsourced study and one in-lab controlled study with 46 participants. We show that the data collected from the crowdsourced experiment accurately follows the results from the in-lab experiments. For Homogeneous Targets the Square-Root model ($T= a + b \sqrt{\frac{A}{W}}$) results in a smaller ERMS error than the two other control models, LOG ($T = a +b\log{\frac{2A}{W}}$) and LOG' ($T = a +b\log{\frac{A}{W}+1}$) for $A/W<10$. Similarly for Heterogeneous Targets the Square-Root model results in a significantly smaller ERMS error when compared to the LOG model for $A/W<10$. The LOG model resulted in significantly smaller ERMS error in the $A/W>15$. In the Heterogeneous Targets the LOG' model consistently resulted in a significantly smaller error for $0

Opinion Space is a system that directly elicits opinions from participants for idea generation. It uses both numerical and textual data and we look at methods to combine these two sets of data. Canonical Correlation Analysis, CCA, is used as a method to combine both the textual and numerical inputs from participants. CCA seeks to find linear transformation matrices that maximize the lower dimension correlation between the projection of numerical rating ($Xw_x$) and textual comments onto the two dimensional space ($Yw_y$). In other words it seeks to solve the following problem $argmax_{w_x,w_y} corr(Xw_x, Yw_y)$ in which $X$ and $Y$ are representations of the numerical rating and textual comments of participants in high dimensions and $Xw_x$ and $Yw_y$ are their lower dimension representations. By using participants' numerical feedbacks on each others' comments, we then develop an evaluation framework to compare different dimensionality reduction methods. In our evaluation framework a dimensionality reduction is the most appropriate for Opinion Space when the value of $\gamma = -corr(r,D)$ has the largest value. In $\gamma = -corr(R,D)$, $R$ is the set of $r_{ij}$ values. $r_{ij}$ is the rating that the participant $i$ is giving to the textual opinion of participant $j$. Similarly $D$ is the set that contains $d_{ij}$ values. $d_{ij}$ is the Euclidean distance between the locations of participant $i$ and $j$. In this dissertation we provide supporting argument as to why this evaluation framework is appropriate for Opinion Space. We have compared different variations of CCA and PCA dimensionality reductions on different datasets. Our results suggests that the $\gamma$ values for CCA are at least $\%169$ larger than the $\gamma$ values of PCA, making CCA a more appropriate dimensionality reduction model for Opinion Space.

A product review on an online retailer website is often accompanied with numerical ratings for the product on different scales, a textual review and sometimes information on whether or not the review is helpful. Generalized Sentiment Analysis looks at the correlation between the textual comment and numerical rating and uses that to infer the numerical ratings on different scales from the textual comment. We provide the formulations for using CCA for solving such a problem. We compare our CCA model with Support Vector Machine, Linear Regression, and other traditional machine learning models and highlight the strengths and weaknesses of this model. We found that training the CCA formulation is significantly faster than SVM which is traditionally used in this context (the fastest training time for SVM in LibSVM was 1,126 seconds while CCA took only 33 seconds for training). We also observed that the Mean Squared Error for CCA was smaller than other competing models (The MSE for CCA with tf-idf features was 1.69 while this value for SVM was 2.28). Linear regression was more sensitive to the featurization method. It resulted in larger MSE when used on multinomial ($MSE = 8.88$)and Bernoulli features ($MSE = 4.21$) but smaller MSE when tf-idf weights were used ($MSE=1.47$).

Robot manipulation research is essential for advancing automation technologies, allowing robotic arms to execute complex and precise tasks across various industries. In recent years, several technologies have matured and transformed industries, such as logistics and manufacturing automation, through advancements like robot grasping. To enable the adoption of these emerging capabilities into real-world automation scenarios, these technologies need to be both efficient and reliable. However, achieving a balance between efficiency and reliability is challenging, as improving one often requires compromising the other. As a result, many impressive methods, initially developed for practical applications, struggle to make the transition into industry use.

This thesis explores five research areas within the field of robot manipulation, examining diverse angles and domains such as industrial automation, deformable manipulation, agricultural robotics, and surgical robotics. It proposes strategies to achieve efficient and reliable robot manipulation. By focusing on enhancing both efficiency and reliability, the thesis aims to facilitate the transition of robot manipulation from proof of concept to practical applications. Among the various methods explored, three primary strategies have proven to be particularly effective: constrained optimization, which given a reliable model, applies strict mathematical constraints to find efficient and reliable solutions; interactive perception and self-supervised learning, which are used to improve the efficiency and reliability in situations where there is high uncertainty in the dynamics of the system or a lack a reliable model. The thesis concludes by discussing the key insights gained through this research, reviewing lessons learned, and suggesting potential directions for future research.

Recent breakthroughs in deep learning have revolutionized natural language processing, computer vision, and robotics. Nevertheless, reliable robot autonomy in unstructured environments remains elusive. Without the Internet-scale data available for language and vision, robotics faces a unique chicken-and-egg problem: robot learning requires large datasets from deployment at scale, but robot learning is not yet reliable enough for deployment at scale. We propose a scalable human-in-the-loop learning paradigm as a potential solution to this paradox, and we argue that it is the key ingredient behind the recent growth of large-scale robot deployments in applications such as autonomous driving and e-commerce order fulfillment. We develop novel formalisms, algorithms, benchmarks, systems, and applications for this setting and evaluate its performance in extensive simulation and physical experiments.

This dissertation is composed of three complementary parts. In Part I, we propose novel algorithms and systems for interactive imitation learning, in which autonomous robots can actively query human supervisors for assistance when needed. In Part II, we introduce interactive fleet learning, which generalizes interactive imitation learning to multiple robots and multiple human supervisors. In Part III, we introduce and study systems for remote supervision of robot fleets over the Internet, enabling interactive fleet learning at a distance. Throughout this thesis, we design algorithms and systems with an emphasis on scalability in terms of the number of robots, number of humans, amount of human supervision required, dataset size, and distribution of physical locations. We conclude with a discussion of limitations and opportunities for future work.

When operating in unstructured and semi-structured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables, where the object may be partially or fully hidden behind other objects. The goal of this task, which we define as mechanical search, is to retrieve a target object in as few actions as possible. Robustly perceiving and manipulating objects is challenging in these scenarios due to the presence of sensor noise, occlusions, and unknown object properties. Because of these perception and manipulation challenges, learning end-to-end mechanical search policies from data is difficult. Instead, we break mechanical search policies into three modules, a perception module that creates an intermediate representation from the input observation, a set of low-level manipulation primitives, and a high-level action selection policy that iteratively chooses which low-level primitives to execute based on the output from the perception module. We explore progress made on manipulation primitives, such as pushing and grasping, segmentation of scenes with unknown objects and occupancy distribution predictions to infer likely locations of the target object. Additionally, we demonstrate that using simulated depth images or point clouds can allow rapid generation of large-scale training datasets for perception networks while allowing them to generalize to real-world objects and scenes. We show that integrating these components can result in an efficient mechanical search policy, improving the success rate by 15% and reducing the number of actions needed to extract the target object as compared to baseline policies in both bin and shelf environments across simulated and physical trials.

A new era of robotic surgery is poised to begin when critical patents held by Intuitive Surgical (IS) expire in 2016. IS market dominance for decades has led to an effective monopoly that will be challenged by several commercial enterprises working on next generation general robotic surgery systems. Robotic surgery has the potential to alleviate the skill-gap between experienced and inexperienced surgeons through the automation of sub-tasks within surgical

procedures.

The primary objective of this dissertation is to explore the process of design itself through the presentation of these devices. The common intention throughout the presented case studies is Frugal Design.

Secondarily, this dissertation explores limitations of current surgical automation and addresses the design considerations of hardware modifcations to the daVinci Research Kit which enable automation.

Part I (Chapter 1) of this dissertation describes the considerations that make a design frugal and the methodology used by the author across three conceptual examples including:

- an inexpensive physical environment for machine learning,

- a device for cloud-enabled irrigation optimization, and

- an interchangeable tool interface for surgical robotics.

Part II of this dissertation presents frugal design case studies for system components that enable an automated silicone-phantom tumorectomy.

Chapter 2 provides an overview of existing technology and related work in the field of surgical automation.

Chapters 3, 4, 5, and 6 describe designs for devices that enable automation of the four subtasks

involved in a tumorectomey: Palpation, Incision, Debridement, and Closure. Specifically, these devices include:

- an inexpensive palpation probe to locate tumors (Chapter 3),

- an interchangeable tool interface for surgical robotics (Chapter 5),

- a fluid injector for surgical robotics (Chapter 4), and

- a passive needle orientation texture (Chapter 6).

Part III of this dissertation describes the design theory behind the author's work presented in Part 1 (examples) and Part 2 (case studies). In Chapter 7, a short history of design theory in the Bauhaus school of (1919-1933) is presented through examples taken from the author's work. Chapter 8 serves as a link to future efforts by the author by bridging the presented frugal design work into the global development effort of frugal innovation.

The design theory material in Part 3 is presented for three reasons:

First: Conceptual design depends on the practitioner's ability to abstract design criteria into diverse functional solutions. The author's practice in this area involves the artistic study of functionalist abstraction as unconstrained exploration of the design space. That practice is presented to elucidate the creative origins of the solutions presented herein, and to provoke creative problem solving in others.

Second: A socially-progressive responsibility, inherent in the Bauhaus ideology, is modernized in Chapter 8 as Frugal Design, which borrows its motivation from a new trend in

development for emerging markets known as Frugal Innovation.

Third: A parallel is drawn to the climate of artificially intelligent automation in present-day Bay Area research in Berkeley, Stanford, and Silicon Valley to the Bauhaus, which was founded as a reaction to increasing automation during the first machine age in Germany.

While humans can grasp and manipulate novel objects with ease, rapid and reliable robot grasping of a wide variety of objects is highly challenging due to sensor noise, partial observability, imprecise control, and hardware limitations.

Analytic approaches to robot grasping use models from physics to predict grasp success but require precise knowledge of the robot and objects in the environment, making them well-suited for controlled industrial applications but difficult to scale to many objects.

On the other hand, deep neural networks trained on large datasets of grasps labeled with empirical successes and failures can rapidly plan grasps across a diverse set of objects, but data collection is tedious, robot-specific, and prone to mislabeling.

To improve the efficiency of learning deep grasping policies, we propose a hybrid method to automate dataset collection by generating millions of synthetic 3D point clouds, robot grasps, and success metrics using analytic models of contact, collision geometry, and image formation.

We present the Dexterity-Network (Dex-Net), a framework for generating training datasets by analyzing mechanical models of contact forces and torques under stochastic perturbations across thousands of 3D object CAD models.

We describe dataset generation models for training policies to lift and transport novel objects from a tabletop or cluttered bin using a 3D depth sensor and a parallel-jaw (two-finger) or suction cup gripper.

To study the effects of learning from massive amounts of training data, we generate datasets containing millions of training examples using distributed Cloud computing, simulations, and parallel GPU processing.

We use these datasets to train robust grasping policies based on Grasp Quality Convolutional Neural Networks (GQ-CNNs) that take as input a depth image and a candidate grasp with up to five degrees of freedom and predict the probability of grasp success on an object in the image.

To transfer from simulation to reality, we develop novel analytic grasp success metrics based on resisting disturbing forces and torques under stochastic perturbations and bounding an object's mobility under an energy field such as gravity.

In addition, we study techniques in algorithmic supervision to guide dataset collection using full knowledge of the object geometry and pose in simulation.

We explore extensions to learning policies that sequentially pick novel objects from dense clutter in a bin and that can rapidly decide which gripper hardware is best in a particular scenario.

To substantiate the method, we describe thousands of experimental trials on a physical robot which suggest that deep learning on synthetic Dex-Net datasets can be used to rapidly and reliably plan grasps across a diverse set of novel objects for a variety of depth sensors, robot grippers, and robot arms.

Results suggest that policies trained on Dex-Net datasets can achieve up to 95% success in picking novel objects from densely cluttered bins at a rate of over 310 mean picks per hour with no additional training or tuning on the physical system.

As an alternative to explicit programming for robots, Deep Imitation learning has two drawbacks: sample complexity and covariate shift. One approach to Imitation Learning is Behavior Cloning, in which a

robot observes a supervisor and then infers a control policy. A known problem with this approach is that even slight departures from the supervisor’s demonstrations can compound over the policy’s roll-out resulting in errors; this concept of drift and resulting error is commonly referred to as covariate shift On-policy techniques reduce covariate shift by iteratively collecting corrective actions for the current robot policy. To reduce sample complexity of these approaches, we propose a novel active learning algorithm, SHIV (Svm-based reduction in Human InterVention). While evaluating SHIV, we reconsider the trade-off between Off- and On-Policy methods and find that: 1) On-Policy methods are challenging for humans supervisors and 2) performance varies with the expressiveness of the policy class. To make Off-Policy methods more robust for expressive policies we propose a second algorithm, DART (Disturbances Augmenting Robot Trajectories), which injects optimized noise into the supervisor’s control stream to simulate error during data collection. This dissertation contributes two aforementioned algorithms, experimental evaluation with three robots evaluating their performance on tasks ranging from grasping in clutter to singulation to bed-making, and the design of a novel first-order urban driving simulator (FLUIDS) that can fill gaps in existing benchmarks for Imitation Learning to rapidly test algorithm performance in terms of generalization.

The goal of this dissertation is to enhance automation in healthcare applications, specifically: brachytherapy delivery for cancer treatment and robot-assisted surgery, used for over 500,000 procedures annually in US alone. Brachytherapy uses cutting-edge computer-assisted planning, but assumes a fixed hardware design. On the contrary, Robot-Assisted Surgery uses high-precision state-of-the-art hardware under complete manual control with little automation. This dissertation is a step towards addressing this gap using a combination of optimization and design. Case studies show that performance of autonomous systems can be improved by leveraging the interaction between optimization based algorithms and the design of hardware systems.

For Brachytherapy, this dissertation has developed a new approach for treatment delivery in Intracavitary Brachytherapy using patient specific 3D printed implants and implemented it on a clinical case of oral cancer. I present an algorithm to quantify reachability with straight-line needles for a given anatomy in prostate cancer. I integrate optimization based needle and dose planning algorithms in interstitial brachytherapy for prostate cancer using two methods of skew-line needle configuration implants: robot-assisted procedures and customized needle guides. The procedures are demonstrated on physical phantoms and performance is compared with an expert physician.

For Robot-Assisted Surgery, this dissertation highlights the interplay between the design of

hardware to reduce uncertainty and optimization based motion planning to enable automated multi-throw suturing . A novel algorithm, Transition State Clustering (TSC) extracts the latent task structure from task demonstrations by segmenting robot trajectories using hierarchical clustering and fitting Gaussian mixture model to identify transitions. I extend TSC with deep learning to perform trajectory segmentation for multi-modal data consisting of kinematics and videos.

The robotics community has seen tremendous advances in grasping and manipulating a wide variety of objects using learning-based techniques. Deep learning approaches are often sample inefficient, and most work deals with grasping rigid objects rather than deformable objects such as ropes, clothing, and bags, which are ubiquitous in our daily lives. In this thesis, I will present novel approaches that can result in more efficient robot learning, including learning from simulation by using a skilled supervisor or building a dynamics model, learning using action-centric neural network architectures, and learning using a curriculum of samples. I will demonstrate results on manipulating deformable objects in simulation and real settings, and learning policies for Atari and MuJoCo environments.