Skip to main content
eScholarship
Open Access Publications from the University of California

Scholarly Works (5 results)

  • Thesis
  • Peer Reviewed
UC Irvine Electronic Theses and Dissertations (2019)

Crowdsourcing is an emerging distributed problem-solving model combining human and machine computation. It collects intelligence and knowledge from a large and diverse workforce to complete complex tasks. In the software engineering domain, crowdsourced techniques have been adopted to facilitate various tasks, such as design, testing, debugging, development, and so on. Specifically, in crowdsourced testing, crowdsourced workers are given testing tasks to perform and submit their feedback in the form of test reports. One of the key advantages of crowdsourced testing is that it is capable of providing engineers software engineers with domain knowledge and feedback from a large number of real users. Based on diverse software and hardware settings of these users, engineers can bugs that are not caught by traditional quality assurance techniques. Such benefits are particularly ideal for mobile application testing, which needs rapid development-and-deployment iterations and support diverse execution environments. However, crowdsourced testing naturally generates an overwhelming number of crowdsourced test reports, and inspecting such a large number of reports becomes a time-consuming yet inevitable task.

This dissertation presents a series of techniques, tools and experiments to assist in crowdsourced report processing. These techniques are designed for improving this task in multiple aspects: 1. prioritizing crowdsourced report to assist engineers in finding as many unique bugs as possible, and as quickly as possible; 2. grouping crowdsourced report to assist engineers in identifying the representative ones in a short time; 3. summarizing the duplicate reports to provide engineers with a concise and accurate understanding of a group of reports;

In the first step, I present a text-analysis-based technique to prioritize test reports for manual inspection. This technique leverages two key strategies: (1) a diversity strategy to help developers inspect a wide variety of test reports and to avoid duplicates and wasted effort on falsely classified faulty behavior, and (2) a risk-assessment strategy to help developers identify test reports that may be more likely to be fault-revealing based on past observations.Together, these two strategies form our technique to prioritize test reports in crowdsourced testing.

Moreover, in the mobile testing domain, test reports often consist of more screenshots and shorter descriptive text, and thus text-analysis-based techniques may be ineffective or inapplicable. The shortage and ambiguity of natural-language text information and the well-defined screenshots of activity views within mobile applications motivate me to propose a novel technique based on using image understanding for multi-objective test-report prioritization.

This technique employs the Spatial Pyramid Matching (SPM) technique to measure the similarity of the screenshots, and apply the natural-language processing technique to measure the distance between the text of test reports.

Next, I design and implement CTRAS: a novel approach to leveraging duplicates to enrich the content of bug descriptions and improve the efficiency of inspecting these reports. CTRAS is capable of automatically aggregating duplicates based on both textual information and screenshots, and further summarizes the duplicate test reports into a comprehensive and comprehensible report.

I validate all of these techniques on industrial data by collaborating with several companies. The results show my techniques can improve both the efficiency and effectiveness of crowdsourced test report processing. Also, I suggest settings for different usage scenarios and discuss future research directions.

    Cover page: Leveraging the Power of Crowds: Automated Test Report Processing for The Maintenance of Mobile ApplicationsCreative Commons 'BY-NC' version 4.0 license
    • Article
    • Peer Reviewed
    UCLA Previously Published Works (2018)
    In many binary classification applications, such as disease diagnosis and spam detection, practitioners commonly face the need to limit type I error (that is, the conditional probability of misclassifying a class 0 observation as class 1) so that it remains below a desired threshold. To address this need, the Neyman-Pearson (NP) classification paradigm is a natural choice; it minimizes type II error (that is, the conditional probability of misclassifying a class 1 observation as class 0) while enforcing an upper bound, α, on the type I error. Despite its century-long history in hypothesis testing, the NP paradigm has not been well recognized and implemented in classification schemes. Common practices that directly limit the empirical type I error to no more than α do not satisfy the type I error control objective because the resulting classifiers are likely to have type I errors much larger than α, and the NP paradigm has not been properly implemented in practice. We develop the first umbrella algorithm that implements the NP paradigm for all scoring-type classification methods, such as logistic regression, support vector machines, and random forests. Powered by this algorithm, we propose a novel graphical tool for NP classification methods: NP receiver operating characteristic (NP-ROC) bands motivated by the popular ROC curves. NP-ROC bands will help choose α in a data-adaptive way and compare different NP classifiers. We demonstrate the use and properties of the NP umbrella algorithm and NP-ROC bands, available in the R package nproc, through simulation and real data studies.
      Cover page: Neyman-Pearson classification algorithms and NP receiver operating characteristics
      • Article
      • Peer Reviewed
      UC Irvine Previously Published Works (2020)
      Open source developers, particularly the elite developers who own the administrative privileges for a project, maintain a diverse portfolio of contributing activities. They not only commit source code but also exert significant efforts on other communicative, organizational, and supportive activities. However, almost all prior research focuses on specific activities and fails to analyze elite developers' activities in a comprehensive way. To bridge this gap, we conduct an empirical study with fine-grained event data from 20 large open source projects hosted on GITHUB. We investigate elite developers' contributing activities and their impacts on project outcomes. Our analyses reveal three key findings: (1) elite developers participate in a variety of activities, of which technical contributions (e.g., coding) only account for a small proportion; (2) as the project grows, elite developers tend to put more effort into supportive and communicative activities and less effort into coding; and (3) elite developers' efforts in nontechnical activities are negatively correlated with the project's outcomes in terms of productivity and quality in general, except for a positive correlation with the bug fix rate (a quality indicator). These results provide an integrated view of elite developers' activities and can inform an individual's decision making about effort allocation, which could lead to improved project outcomes. The results also provide implications for supporting these elite developers.
        Cover page: Unveiling Elite Developers’ Activities in Open Source ProjectsCreative Commons 'BY' version 4.0 license
        • Article
        • Peer Reviewed
        LBL Publications (2022)
        In this study, we detect and track extratropical cyclones using 6-hourly mean sea level pressure data taken from the Twentieth Century Reanalysis version 3 (20CRv3) over the period 1951–2015 and compare them with those in the Interim and fifth generation of ECMWF reanalyses over the period 1979–2018. Three indices were employed to characterize cyclone activity, including cyclone count, cyclone intensity, and a cyclone activity index (CAI) that combines the count and intensity. The results show that the cyclone indices in the three datasets have comparable annual climatologies and seasonal evolution over the northern extratropical land and ocean in recent decades. Based on the cyclone indices over the period 1951–2010 in 80 ensemble members of 20CRv3, cyclone count and intensity are negatively correlated in winter and tend to be positively and weakly correlated in summer. The interannual CAI variability is dominated by the cyclone count variability. Regional mean cyclone activity can be well represented using the ensemble average cyclone index. We then examined the linkage of the cyclone activity in 20CRv3 and observed cold and warm extremes over Eurasia and North America over the period 1951–2010. In winter, the principal components of interannual cold and warm extreme anomalies are more correlated with the regional mean cyclone count index over Eurasia, while they are more correlated with the cyclone intensity index over North America. The temperature anomalies associated with the regional and ensemble mean cyclone count index explain about 10% (20%) of interannual cold (warm) extreme variances averaged over Eurasia. The temperature anomalies associated with the mean cyclone intensity explain about 10% of interannual cold and warm extreme variances over North America. Large-scale atmospheric circulation anomalies in association with cyclone activity and the induced temperature advection drive temperature anomalies over Eurasia and North America. In summer, circulation and thermal advection anomalies associated with cyclone activity are weak over the two continents. Hence, that season’s relationship between cyclone activity and extreme temperature variability is weak.
          Cover page: Northern Hemisphere Extratropical Cyclone Activity in the Twentieth Century Reanalysis Version 3 (20CRv3) and Its Relationship with Continental Extreme TemperaturesCreative Commons 'BY' version 4.0 license
          • Article
          • Peer Reviewed
          LBL Publications (2020)
          This dataset, produced through the Coordinated Ocean Wave Climate Project (COWCLIP) phase 2, represents the first coordinated multivariate ensemble of 21st Century global wind-wave climate projections available (henceforth COWCLIP2.0). COWCLIP2.0 comprises general and extreme statistics of significant wave height (HS), mean wave period (Tm), and mean wave direction (θm) computed over time-slices 1979-2004 and 2081-2100, at different frequency resolutions (monthly, seasonally and annually). The full ensemble comprising 155 global wave climate simulations is obtained from ten CMIP5-based state-of-the-art wave climate studies and provides data derived from alternative wind-wave downscaling methods, and different climate-model forcing and future emissions scenarios. The data has been produced, and processed, under a specific framework for consistency and quality, and follows CMIP5 Data Reference Syntax, Directory structures, and Metadata requirements. Technical comparison of model skill against 26 years of global satellite measurements of significant wave height has been undertaken at global and regional scales. This new dataset provides support for future broad scale coastal hazard and vulnerability assessments and climate adaptation studies in many offshore and coastal engineering applications.
            Cover page: A global ensemble of ocean wave climate projections from CMIP5-driven models
            Top