UC San Diego

Large Language Models~(LLMs) have demonstrated remarkable progress across diverse natural language tasks. Recently, Chain-of-Thought methods have been introduced to enhance reasoning by generating detailed and comprehensive reasoning processes. However, challenges such as hallucinations, error accumulation, and limited exploration hinder their effectiveness on complex tasks. Additionally, the near-exhaustion of high-quality natural language data on the internet poses a significant barrier to further improving the reasoning capabilities of LLMs.

To address these challenges, this dissertation investigates two key directions: enhancing inference techniques and synthesizing reasoning data to strengthen LLM reasoning capabilities. First, it introduces a deductive verification method that enables self-verification of reasoning chains generated by LLMs, ensuring more rigorous and accurate reasoning during inference. Second, it addresses the limitations in exploring diverse reasoning strategies by framing reasoning as a hierarchical policy, where high-level tactics guide detailed low-level problem-solving through in-context learning with LLMs. In addition, it explores data synthesis for long-context reasoning tasks, which is particularly challenging even for human and very rare natural data on the internet. It proposes a novel data synthesis method that can generate long-context reasoning data with diverse and realistic reasoning patterns. The evaluation of the generated long-context reasoning dataset using this method reveals that even state-of-the-art LLMs struggle to perform robustly, highlighting the potential of the synthetic data strategy for enhancing LLM training. This dissertation contributes to advancing LLM reasoning abilities through novel methods that address critical limitations in both training and inference. These advancements provide valuable insights and pave the way for stronger and more reliable reasoning in LLMs.