We present an initial study of off-policy evaluation (OPE), a problem prerequisite to real-world reinforcement learning (RL), in the context of building control. OPE is the problem of estimating a policy's performance without running it on the actual system, using historical data from the existing controller. It enables the control engineers to ensure a new, pretrained policy satisfies the performance requirements and safety constraints of a real-world system, prior to interacting with it. While many methods have been developed for OPE, no study has evaluated which ones are suitable for building operational data, which are generated by deterministic policies and have limited coverage of the state-action space. After reviewing existing works and their assumptions, we adopted the approximate model (AM) method. Furthermore, we used bootstrapping to quantify uncertainty and correct for bias. In a simulation study, we evaluated the proposed approach on 10 policies pretrained with imitation learning. On average, the AM method estimated the energy and comfort costs with 1.84% and 14.1% error, respectively.

In this paper, we use data collected from over 2000 non-residential electric vehicle supply equipments (EVSEs) located in Northern California for the year of 2013 to estimate the potential benefits of smart electric vehicle (EV) charging. We develop a smart charging framework to identify the benefits of non-residential EV charging to the load aggregators and the distribution grid. Using this extensive dataset, we aim to improve upon past studies focusing on the benefits of smart EV charging by relaxing the assumptions made in these studies regarding: (i) driving patterns, driver behavior and driver types; (ii) the scalability of a limited number of simulated vehicles to represent different load aggregation points in the power system with different customer characteristics; and (iii) the charging profile of EVs. First, we study the benefits of EV aggregations behind-the-meter, where a time-of-use pricing schema is used to understand the benefits to the owner when EV aggregations shift load from high cost periods to lower cost periods. For the year of 2013, we show a reduction of up to 24.8% in the monthly bill is possible. Then, following a similar aggregation strategy, we show that EV aggregations decrease their contribution to the system peak load by approximately 37% (median) when charging is controlled within arrival and departure times. Our results also show that it could be expected to shift approximately 0.25. kW. h (~2.8%) of energy per non-residential EV charging session from peak periods (12. PM-6. PM) to off-peak periods (after 6. PM) in Northern California for the year of 2013.

Human-Building Interaction (HBI) is a convergent field that represents the growing complexities of the dynamic interplay between human experience and intelligence within built environments. This paper provides core definitions, research dimensions, and an overall vision for the future of HBI as developed through consensus among 25 interdisciplinary experts in a series of facilitated workshops. Three primary areas contribute to and require attention in HBI research: humans (human experiences, performance, and well-being), buildings (building design and operations), and technologies (sensing, inference, and awareness). Three critical interdisciplinary research domains intersect these areas: control systems and decision making, trust and collaboration, and modeling and simulation. Finally, at the core, it is vital for HBI research to center on and support equity, privacy, and sustainability. Compelling research questions are posed for each primary area, research domain, and core principle. State-of-the-art methods used in HBI studies are discussed, and examples of original research are offered to illustrate opportunities for the advancement of HBI research.

As buildings account for approximately 40% of global energy consumption and associated greenhouse gas emissions, their role in decarbonizing the power grid is crucial. The increased integration of variable energy sources, such as renewables, introduces uncertainties and unprecedented flexibilities, necessitating buildings to adapt their energy demand to enhance grid resiliency. Consequently, buildings must transition from passive energy consumers to active grid assets, providing demand flexibility and energy elasticity while maintaining occupant comfort and health. This fundamental shift demands advanced optimal control methods to manage escalating energy demand and avert power outages. Reinforcement learning (RL) emerges as a promising method to address these challenges. In this paper, we explore ten questions related to the application of RL in buildings, specifically targeting flexible energy management. We consider the growing availability of data, advancements in machine learning algorithms, open-source tools, and the practical deployment aspects associated with software and hardware requirements. Our objective is to deliver a comprehensive introduction to RL, present an overview of existing research and accomplishments, underscore the challenges and opportunities, and propose potential future research directions to expedite the adoption of RL for building energy management.