How microorganisms respond to and interact with their environment can vary significantly from individual to individual, which can have important microbiological and ecological implications. However, most microscopy techniques can only observe motile microorganisms for short times because of their limited fields of view. Using Lagrangian tracking, a single microorganism can be followed in 3D, potentially indefinitely, allowing to decipher individual phenotypical traits. Current Lagrangian tracking methods use the fluorescence signal emitted by the microorganism as feedback to keep it in focus. However, over long times, epifluorescent imaging can induce photobleaching and photodamage, and importantly, not all microorganisms can easily be made fluorescent. Additionally, traditional algorithms used in feedback loops to determine microorganism position are prone to errors, especially in optically complex media. Here, we present a faster, more reliable, and versatile Lagrangian tracking method that uses deep learning to determine the 3D position of the microorganism. This new method demonstrates enhanced accuracy and speed in tracking fluorescent bacteria with fluorescence microscopy also in optically complex media. Furthermore, we track bacteria with other microscopy modalities, such as brightfield microscopy -- for example, this enables us to track magnetotactic bacteria, which cannot be made fluorescent without degrading their magnetotactic properties. These novel capabilities allow to extract previously inaccessible quantitative information, significantly advancing the study of microorganism behavior -- and thus opening new avenues for research in complex biological and ecological systems.