Applying data analytics in production test has become a widely adopted industrial practice in recent years. As the complexity of semiconductor devices scales and the amounts of available test data continue to grow, the research direction in this field is forced to shift away from solving specific problems with ad hoc approaches and demands for deeper understanding of the fundamental issues. Two data-driven test applications where this shift is apparent are production yield optimization and defect screening, where the respective underlying data analytics approaches are correlation analysis and outlier analysis. A core issue present in these two approaches stems from the subjectivity that is inherent to data analytics. This dissertation delves into how subjectivity manifests itself and what can be done to reduce it with respect to the two test applications.

Outlier analysis is an approach used for identifying anomalies. The main goal of outlier analysis in test is to capture statistically outlying parts with the hope that their abnormal behavior is attributed to some defectivity. During creation of an outlier model, the decisions about outlying behavior in the existing data are made by utilizing known failures and the test engineer's best judgment. In practice, outlier screening methods are simply used for transforming data into an outlier score space. Even if outlier analysis techniques are able to successfully classify a dataset into inliers and outliers, outlier models require thresholds to be decided. A concept called Consistency is introduced to provide an objective data-driven way to evaluate outlier models by utilizing all available data. The key observation underlying this concept is that outlier analysis should be immune to noise introduced by sources of systematic variation.

Correlation analysis is a process comprising a search for related variables. The application of production yield optimization involves searching for correlation between the yield and various controllable parameters. The goal of this process is to uncover parameters that, when adjusted, can result in yield improvement. This analytics process is subjective to the perspective of the analyst and the quality of the result is highly dependent on the analyst’s previous experiences. In order to reduce the subjectivity in this application, a process mining methodology is introduced to learn from the experiences of analysts. The key advantage of this methodology is that in addition to having the capability to record and reproduce these analyses, it can also generalize to analytics processes not contained in the learned experiences.

Functional verification continues to be one of the most time-consuming steps in the chip design cycle. Simulation-based verification is well practised in industry thanks to its flexibility and scalability. The completeness of the verification is measured by coverage metrics. Generating effective tests to achieve a satisfactory coverage level is a difficult task in verification. Constrained random verification is commonly used to alleviate the manual efforts for producing direct tests. However, there are yet many situations where unnecessary verification efforts in terms of simulation cycles and man hours are spent. Also, it is observed that lots of data generated in existing constrained random verification process are barely analysed, and then discarded after simplistic correctness checking. Based on our previous research on data mining and exposure to the industrial verification process, we identify that there are opportunities in extracting knowledge from the constrained random verification data and use it to improve the verification efficiency.

In constrained random verification, when a simulation run of tests instantiated by a test template cannot reach the coverage goal, there are two possible reasons: insufficient simulation, and improper constraints and/or biases. There are three actions that a verification engineer can usually do to address the problem: to simulate more tests, to refine the test template, or to change to a new test template. Accordingly, we propose three data learning methodologies to help the engineers make more informed decisions in these three application scenarios and thus improve the verification efficiency.

The first methodology identifies important ("novel") tests before simulation based on what have been already simulated. By only simulating those novel tests and filtering out redundant tests, tremendous resources such as simulation cycles and licenses can be saved. The second methodology extracts the unique properties from those novel tests identified in simulation and uses them to refine the test template. By leveraging the extracted knowledge, more tests similar to the novel ones are generated. And thus the new tests are more likely to activate coverage events that are otherwise difficult to hit by extensive simulation. The third methodology analyses a collection of existing test items (test templates) and identifies feasible augmentation to the test plan. By automatically adding new test items based on the data analysis, it alleviates the manual efforts for closing coverage holes.

The proposed data learning methodologies were developed and applied in the setting of verifying commercial microprocessor and SoC platform designs. The experiments in this dissertation were conducted in the verification environment of a commercial microprocessor and a SoC platform in Freescale Semiconductor Inc. and were in parallel with the on-going verification efforts. The experiment results demonstrate the feasibility and effectiveness of building learning frameworks to improve verification efficiency.

This work explores Domain Specific Machine Learning (DSML) problems that arise from a workflow in a semiconductor company. The focus is on understanding what characteristics make them different from a typical machine learning problem. We first discuss a general setup for a DSML problem and provide two postulates to describe the scope of the DSML considered in this research. We examine the theories of machine learning, including four different schools of thinking for articulation of what machine learning is. We review the No Free Lunch Theory for machine learning and discuss the incompleteness of the four theories in view of the No Free Lunch.

Based on the theoretical understanding, we then point out the essence of DSML.Specifically, a DSML problem considered in this work follows an iterative process. In each iteration, the learning goal is to attain a ``surprise'' for the next iteration. Essentially, the unpredictability in the data appears in the next iteration represents the value of the learning. From this point of view, we argue that the underlying problem in DSML is local No Free Lunch.

An important consequence of this view is that machine learning will no longer be used in making a decision for the user, as that traditionally used. Instead, it is used to help a data summarization process to facilitate user to make a decision. In this thesis, we investigate two types of applications, outlier screening in production testing and wafer map pattern recognition for yield monitoring and improvement. We use these two applications to illustrate our view of DSML as well as various considerations associated with the view. Based on experience of implementing a machine learning software solution in a company, we explain the lessons learned for deploying a successful machine learning solution. If we consider traditional machine learning as largely focusing on the learning algorithms, we conclude that DSML is largely focusing on all other aspects trying to leverage the capability of a learning algorithm (if possible). From this angle, DSML is solving a problem complementary to machine learning (Co-ML). In other words, DSML shall not be taken as another direct extension of ML. Instead, it would be more appropriate to view DSML (in the semiconductor industry) as Co-ML.

This dissertation focuses on Domain-Specific Machine Learning (DSML) with applications in the semiconductor industry. We first illustrate characteristics of DSML in view of semiconductor design and test processes, where a task is usually carried out in an iterative fashion. We introduce a new concept called Local No-Free-Lunch (Local NFL) to characterize the learning problem in an iteration, where one learns from the data in the current iteration and tries to predict the outcome in the next iteration. A consequence of Local NFL is that optimization of a learning model is no longer as critical in DSML as that in traditional Machine Learning.

To study how to build a practical DSML solution, we take Wafer Map Pattern Recognition (WMPR) as the application context. Historically, WMPR is an important problem to solve for semiconductor manufacturers, where wafer failing patterns are used to guide their yield improvement effort. In recent years, fabless companies became interested in the problem because understanding wafer patterns can help them communicate more effectively with the manufacturers. In the past, prior works treated WMPR as solving a multi-class classification problem. However, in view of DSML, we argue that a multi-class classifier is not what is needed for building a practical DSML solution.

We then develop the requirements and constraints for building a practical DSML solution. From there, we reach a list of tool capabilities to be supported by the solution. To enable these capabilities, we develop novel learning techniques for performing various analyses on wafer maps. Based on the enabled capabilities, a novel methodology is introduced. Given a sequence of wafer maps, this methodology enables a user to quickly see what potential patterns are in the sequence and based on their own perspective, to make a final decision to classify them.

This methodology enables building a practical DSML solution that helps a user answer two main questions in each iteration: (1) Is there a wafer pattern indicating a potential yield issue? 2) If yes, has this pattern occurred previously? An important aspect of the solution is that it enables a user to quickly grasp what crucial information is contained in the data and facilitates them to make a final classification decision. The solution does not make a final decision for them. From this angle, the focus of a DSML solution is on understanding of the data, rather than on optimization of a learning model.

The yield optimization data analytic process is an iterative search process. Each iteration

comprises of two main steps: data preparation, and result evaluation. In this thesis, we

focus on the result evaluation step, where the analyst evaluates the result from running

a certain analytic task looking for some specic concept. For example this concept can

be in the form a pattern formed by the failing dies on a wafer.

A wafer pattern may hint a particular issue in the production by itself or guide the

analysis into a certain direction. In this thesis, we show how Generative Adversarial

Networks (GANs) can be used to build concept recognizers, we present the architecture

chosen for the convolutional neural networks, we show how the network is trained, and

we show how the discriminator network can be used for concept recognition.

However, the main focus of this work is on rules of Tensor decomposition in the

automated concept recognition

ow. We introduce a novel concept recognition approach

based on Tucker decomposition. Tensor-based concept recognizers are combined with

GANs-based recognizers to prevent escapes, also known as adversarial examples.

In this work, we are concerned with two main aspects: (1) automation of the concept

recognition process, and (2) ensuring robustness. Two tensor methods are introduced to

address both aspects.

The rst tensor method is based on clustering and is used to automatically extract

concepts that might be of interest to the analyst as well as choose the set of wafers to be

used in training for each concept.

Our second tensor method is to ensure robustness of the GANs-based recognizers

by introducing the containment check to continuously check for GANs performance and

report to the expert if a problem occurs.

We also address other challenges, such as automating the training process for GANs.

We also introduce a collaboration view where the machine learning expert is assumed

to be a separate entity. Hence he is not allowed to have access to protected information

such as yield. we present a tensor-based transformation for training wafers such that the

protected information is hidden.

Our automated and robust software is applied to two high reliability automotive

products, with 8300 and 7052 wafers respectively.

The emergence of Large Language Models (LLMs) offers new opportunities for applying Machine Learning (ML) and Artificial Intelligence (AI) in semiconductor chip design and test (D&T). Realizing these opportunities requires a fundamentally different thinking from the past. For more than two decades, the semiconductor industry has been exploring applications of ML in D&T. Despite many promises, notable challenges remain in most of the application contexts.

The first part of the thesis (Chapter 2, 3 and 4) includes a review of works for applying ML in D&T, starting in 2003, and describes the journey leading to the current development of an AI Assistant called Intelligent Engineering Assistant (IEA). The journey evolved from one view to another, where each view perceived applying ML in D&T differently. In the first decade, the research took a data-driven view similar to that in common ML practices. This view was changed to a knowledge-driven view in 2014, due to the experience of solving a production yield problem for an automotive chip supplycompany. This experience is highlighted in the thesis, together with learning lessons from a variety of other works in the first decade.

The knowledge-driven view then lasted for four years. During the period, the research focused on finding ways to incorporate domain knowledge into the data learning process. It was in this period, the idea of Co-ML (complementary ML) first emerged. Co-ML formulates a ML problem as a decision problem where the outcome of the learning can be either a model (an answer) or no model (no answer). Then, in 2018 the idea of IEA, as an autonomous system, was first construed. This changed the knowledge-driven view to an autonomous system view.

In 2022, the autonomous system view was once again revised. It was realized that in order to achieve a practical IEA, one had to take a fundamentally different view from the past and perceived problem and solution as a pair, rather than perceived problem as given for finding a solution. We call this thinking the problem-solution dual view (Chapter 5).

Under our problem-solution dual view, applying ML in D&T is no longer seen as “static” in the sense that for a given problem, here is an ML tool for it. It is seen as a data exploration process, a search process driven by user, where each search step comprises a pair of problem instruction and problem solver. Consequently, there are two requirements for an IEA: (1) to provide a language for specifying problem instructions and (2) to provide a software platform capable of solving each acceptable problem instruction. This novel IEA thinking was realized in our first end-to-end IEA in 2022 (IEA-2022) inthe application context of wafermap analytics.

The development of IEA-2022 preceded the release of ChatGPT. At the time, IEA-2022 utilizes its predecessor GPT-3 model, only in a restricted way because of the limitations of the LLM. Then, the release of ChatGPT and its later models fundamentally changed design of IEA again (Chapter 6). In the latest IEA, called IEA-Plot, a knowledge graph (KG) is in place as the central piece to connect problem instruction to problem solver (Chapter 7). With a powerful LLM, the problem instruction can therefore be given in natural language. The instruction is then grounded by the KG internal to IEA in order to find a matching solution based upon a collection of solvers in the backend of IEA. IEA-Plot, again focusing on wafermap analytics, was demonstrated based on test data collected from several product lines. In the thesis, four chapters are devoted to discuss the development of IEA-Plot which is the central piece of this thesis work.

IEA-Plot is the first step moving forward to build a practical LLM-assisted AI Assistant in the semiconductor domain. There is one essential issue with the development of IEA-Plot: the construction of the KG. This motivated us to explore the possibility of using an LLM to assist the development of KG. Furthermore, in the current IEA the domain knowledge is stored explicitly in the KG. The LLM is used off-the-shelf. This raises the question whether or not it is possible to ingest the domain knowledge into an LLM and remove the dependency on KG. The last part (Chapter 8) of the thesis will touchbase on these two aspects. In particular, we present a novel idea called oracle-checker scheme (OC scheme) for utilizing an LLM by treating the LLM as an oracle. Findings for using LLM for KG development are summarized. Then, in the last Chapter 9 before the conclusion we paint a picture for how to ingest domain knowledge into an LLM by taking a generative AI approach.

While IEA-Plot is at the center of this thesis, it should not be seen as a standalone invention. IEA-Plot is a direct consequence from two decades of research on trying to apply ML in D&T. It exemplifies how to build an LLM-assist AI Assistant in practice. It marks the end of a two-decade journey and opens a new one toward generative AI.

We consider the problem of answering queries using views, where queries and views are conjunctive queries with arithmetic comparisons over dense orders. Previous work only considered limited variants of this problem, without giving a complete solution. We first show that obtaining equivalent rewritings for conjunctive queries with arithmetic comparisons is decidable. Then, we consider the problem of finding maximally contained rewritings (MCRs) where the decidability proof does not carry over. We investigate two special cases of this problem where the query uses only semi-interval comparisons. In both cases decidability of finding MCRs depends on the query containment test. First, we address the case where the homomorphism property holds in testing query containment. In this case decidability is easy to prove but developing an efficient algorithm is not trivial. We develop such an algorithm and prove that it is sound and complete. This algorithm applies in many cases where the query uses only left (or right) semi-interval comparisons. Then, we develop a new query containment test for the case where the containing query uses both left and right semi-interval comparisons but with only one left (or right) semi-interval subgoal. Based on this test, we show how to produce an MCR which is a Datalog query with arithmetic comparisons. The containment test that we develop obtains a result of independent interest. It finds another special case where query containment in the presence of arithmetic comparisons can be tested in nondeterministic polynomial time. (c) 2006 Elsevier B.V. All rights reserved.

In this paper, we study the following problem. Given a database and a set of queries, we want to find a set of views that can compute the answers to the queries, such that the amount of space, in bytes, required to store the viewset is minimum on the given database. (We also handle problem instances where the input has a set of database instances, as described by an oracle that returns the sizes of view relations for given view definitions.) This problem is important for applications such as distributed databases, data warehousing, and data integration. We explore the decidability and complexity of the problem for workloads of conjunctive queries. We show that results differ significantly depending on whether the workload queries have self-joins. Further, for queries without self-joins we describe a very compact search space of views, which contains all views in at least one optimal viewset. We present techniques for finding a minimum-size viewset for a single query without self-joins by using the shape of the query and its constraints, and validate the approach by extensive experiments.

Design verification has been a challenging problem due to the increasing complexity of modern system-on-chip (SoC) designs and it is considered one of the costliest processes in hardware design flow. This dissertation investigates a major labor-intensive task, generating tests to hit a given coverage point, in simulation-based verification, and proposes an autonomous software system capable of completing the task. A key feature of the proposed system is its learning capability -- it can learn from examples provided by human engineers to improve itself. There are three major components in the proposed system: test generation, a knowledge database, and rule learning algorithms. The proposed system is able to retrieve information from the database, use the information to analyze simulation results, and generate new tests based on the analysis. Several machine learning techniques are used in the proposed system. For test generation, a novel method, constrained process discovery, is used to learn a test case generation model from manually developed tests. The test case generation model can create new tests and increase its test generation capability by learning from tests developed by humans. For creating a knowledge database, text mining methods are used to extract important design features from design documents. Experiments showed that the extracted signals can be utilized as observation points to infer important hardware events. Last, a novel rule learning method, VeSC-CoL, is proposed to analyze simulation results. VeSC-CoL can handle extremely imbalanced data, which is common in verification, while traditional rule learning methods cannot.