For many of the physical phenomena around us, we have developed sophisticated\nmodels explaining their behavior. Nevertheless, measuring physical properties\nfrom visual observations is challenging due to the high number of causally\nunderlying physical parameters -- including material properties and external\nforces. In this paper, we propose to measure latent physical properties for\ncloth in the wind without ever having seen a real example before. Our solution\nis an iterative refinement procedure with simulation at its core. The algorithm\ngradually updates the physical model parameters by running a simulation of the\nobserved phenomenon and comparing the current simulation to a real-world\nobservation. The correspondence is measured using an embedding function that\nmaps physically similar examples to nearby points. We consider a case study of\ncloth in the wind, with curling flags as our leading example -- a seemingly\nsimple phenomena but physically highly involved. Based on the physics of cloth\nand its visual manifestation, we propose an instantiation of the embedding\nfunction. For this mapping, modeled as a deep network, we introduce a spectral\nlayer that decomposes a video volume into its temporal spectral power and\ncorresponding frequencies. Our experiments demonstrate that the proposed method\ncompares favorably to prior work on the task of measuring cloth material\nproperties and external wind force from a real-world video.\n