Recent advances in technologies for cheaper and faster data acquisition and storage have led to an explosive growth of data complexity in a variety of scientific areas. As a result, noise accumulation, experimental variation, and data inhomogeneity have become substantial. However, many classical classification and regression methods in such settings are known to pose many statistical challenges and hence call for new methods and theories.

This thesis is devoted to robust classification and regression algorithms with theoretical guarantee on important statistical properties. In Chapter 1, we present a classification framework – ArchBoost, which applies to a wide range of loss functions including nonconvex losses and is specifically designed to be robust and efficient whenever the labels are recorded with an error or whenever the data are contaminated with outliers. In Chapter 2, we introduce a forest-type framework for regression problems, and prove that many state-of-the art forest algorithms belong to this framework. We then propose robust forest-type regression methods by applying our proposed framework to robust loss functions. In Chapter 3, we design a novel estimating equation motivated by the framework in Chapter 2 to solve quantile regression problem on random censored data. In Chapter 4, we focus on high-dimensional left-censored quantile regression and study its inference problem. We modify the quantile loss to accommodate the left-censored nature of the problem, by extending the idea of redistribution of mass. For the inference part, asymptotic properties are carefully investigated. All the methods in aforementioned chapters are tested through extensive numerical experiments on both simulated and real data sets.

The first two chapters consider the estimation and inference of the dynamic treatment effect when the confounders are possibly high dimensional. Chapter 1 proposes a sequential doubly robust Lasso (S-DRL) estimator using $\ell_1$-regularized nuisance estimates with DR-type imputations. The proposed method achieves consistency as long as at least one nuisance function is appropriately parametrized for each exposure time and treatment path. The key to achieving these results is the usage of DR representations for intermediate conditional outcome models, which offer superior inferential performance while requiring weaker assumptions. We establish root-n inference based on the S-DRL estimator is guaranteed when two product-sparsity conditions are satisfied. Chapter 2 further provides root-$n$ inference for the dynamic treatment effect even when model misspecification occurs. We provide valid inference based on a ``sequential model double robust'' solution as long as one of the nuisance models is correctly specified at each time spot. Chapter 3 proposes a novel construction for random forests, incorporating cyclic modification of the selection of splitting directions with the goal of achieving a faster consistency rate for the integrated mean squared error (IMSE). Setting $\alpha=0.5$ leads to the proposed cyclic forest degenerating into cyclic median forests, obtaining a minimax optimal rate for IMSE within the Lipschitz class. We further extend our exploration to local polynomial regression within each leaf, formulating cyclic local polynomial forests as generalizations of the cyclic forests. When $\alpha=0.5$, our cyclic local polynomial forests attain a minimax rate for IMSE, marking the first instance of achieving minimax optimal rates for random forests within the H\" older class. Furthermore, we establish minimax optimal rates for the uniform convergence rate.

In the first two chapters, we consider inference for high-dimensional left-censored linear models. Left-censored data arises from measurement limits in scientific devices and social science data. We consider the problem of constructing confidence intervals for the parameters in left-censored linear models. In Chapter 1, we present smoothed estimating equations (SEE) and smoothed robust estimating equations(SREE) frameworks that are adaptive to censoring level and are more robust to misspecification of the error distribution. In Chapter 2, we study inference problem for parameters in high-dimensional left-censored quantile regression model. We modify the quantile loss to accommodate the left-censored nature of the problem, by extending the idea of redistribution of mass. Furthermore, applying the de-biasing technique to the initial estimator leads to an improved estimator suitable for high-dimensional inference under left-censored quantile regression setting. For both problems, asymptotic properties have been investigated.

In Chapter 3, we devise a projection pursuit testing procedure for generalized hypotheses on high-dimensional precision matrix. We illustrate the procedure under specific examples of hypotheses: testing for row sparsity, minimum signal strength, bandedness and generalized bandedness. We demonstrate the performance of the testing procedure through extensive numerical experiments, and present the findings for two real datasets.