REAL-TIME MOTION MONITORING USING DEEP LEARNING

Systems and methods may be used for estimating instantaneous patient motion (a patient state). The patient state may be estimated based on a 3D reference volume and a stream of images, for example from an image acquisition device. The stream of images may be received in real-time, for example during a radiation therapy treatment. An example method may include encoding the 3D reference volume using a 3D encoder branch of a patient state generator network, encoding the stream of images using a 2D encoder branch of the patient state generator network, and combining the encoded 3D reference volume and the encoded real-time stream of images. The method may include estimating a 3D spatial transform that maps the 3D reference volume to a current patient state by decoding the combined encoding using a 3D decoder branch of the patient state generator network.