Longitudinal studies are vital to understanding dynamic changes of the\nplanet, but labels (e.g., buildings, facilities, roads) are often available\nonly for a single point in time. We propose a general model, Temporal Cluster\nMatching (TCM), for detecting building changes in time series of remotely\nsensed imagery when footprint labels are observed only once. The intuition\nbehind the model is that the relationship between spectral values inside and\noutside of building's footprint will change when a building is constructed (or\ndemolished). For instance, in rural settings, the pre-construction area may\nlook similar to the surrounding environment until the building is constructed.\nSimilarly, in urban settings, the pre-construction areas will look different\nfrom the surrounding environment until construction. We further propose a\nheuristic method for selecting the parameters of our model which allows it to\nbe applied in novel settings without requiring data labeling efforts (to fit\nthe parameters). We apply our model over a dataset of poultry barns from\n2016/2017 high-resolution aerial imagery in the Delmarva Peninsula and a\ndataset of solar farms from a 2020 mosaic of Sentinel 2 imagery in India. Our\nresults show that our model performs as well when fit using the proposed\nheuristic as it does when fit with labeled data, and further, that supervised\nversions of our model perform the best among all the baselines we test against.\nFinally, we show that our proposed approach can act as an effective data\naugmentation strategy -- it enables researchers to augment existing structure\nfootprint labels along the time dimension and thus use imagery from multiple\npoints in time to train deep learning models. We show that this improves the\nspatial generalization of such models when evaluated on the same change\ndetection task.\n