Computer vision and robotics are being increasingly applied in medical\ninterventions. Especially in interventions where extreme precision is required\nthey could make a difference. One such application is robot-assisted retinal\nmicrosurgery. In recent works, such interventions are conducted under a\nstereo-microscope, and with a robot-controlled surgical tool. The\ncomplementarity of computer vision and robotics has however not yet been fully\nexploited. In order to improve the robot control we are interested in 3D\nreconstruction of the anatomy and in automatic tool localization using a stereo\nmicroscope. In this paper, we solve this problem for the first time using a\nsingle pipeline, starting from uncalibrated cameras to reach metric 3D\nreconstruction and registration, in retinal microsurgery. The key ingredients\nof our method are: (a) surgical tool landmark detection, and (b) 3D\nreconstruction with the stereo microscope, using the detected landmarks. To\naddress the former, we propose a novel deep learning method that detects and\nrecognizes keypoints in high definition images at higher than real-time speed.\nWe use the detected 2D keypoints along with their corresponding 3D coordinates\nobtained from the robot sensors to calibrate the stereo microscope using an\naffine projection model. We design an online 3D reconstruction pipeline that\nmakes use of smoothness constraints and performs robot-to-camera registration.\nThe entire pipeline is extensively validated on open-sky porcine eye sequences.\nQuantitative and qualitative results are presented for all steps.\n