In this paper, we aim to automate the adjustment of air handling unit (AHU) setpoints within heating, ventilation, and air conditioning (HVAC) systems to maintain indoor temperatures at user-specified levels. A key challenge lies in obtaining sufficient high-quality sensor data from real buildings. To address this, we explore transfer learning and leverage simulation software to generate training data. We propose an adaptive model-based transfer learning approach for dynamic HVAC control, where the agent directly controls the source domain under conditions identical to the target domain. This eliminates the need for extensive target-specific knowledge to define data generation schedules and reduces the risk of collecting irrelevant samples, while also providing greater flexibility during learning. At the control level, we enhance performance through physics rule embedding, which ensures physical consistency, and long-term-aware setpoint selection strategy, which mitigates abrupt setpoint changes. Finally, to accelerate and stabilize deployment in new buildings, we enable knowledge transfer directly between similar real-world buildings, reducing the need to construct virtual source domains repeatedly. Note to Practitioners—Heating, ventilation, and air conditioning (HVAC) systems are major contributors to building energy use, yet their operation often relies on fixed schedules or manually tuned controllers that are slow to adapt and may compromise comfort or efficiency. A key obstacle to deploying more advanced data-driven methods is the scarcity of high-quality sensor data in real buildings. To address this, we propose an adaptive transfer learning approach that first trains an HVAC control agent in a digital twin environment and then fine-tunes it with only a small amount of building-specific data. By embedding simple physical laws and using a long-term control strategy, the method avoids abrupt setpoint changes that lead to equipment wear and occupant discomfort. In practice, this enables faster deployment of intelligent HVAC control with reduced data requirements, yielding smoother indoor temperatures and energy savings. While unusual buildings may still require additional customization, the approach scales well across similar facilities, supporting efficient expansion of smart building management.