Multi-object tracking is an important ability for an autonomous vehicle to\nsafely navigate a traffic scene. Current state-of-the-art follows the\ntracking-by-detection paradigm where existing tracks are associated with\ndetected objects through some distance metric. The key challenges to increase\ntracking accuracy lie in data association and track life cycle management. We\npropose a probabilistic, multi-modal, multi-object tracking system consisting\nof different trainable modules to provide robust and data-driven tracking\nresults. First, we learn how to fuse features from 2D images and 3D LiDAR point\nclouds to capture the appearance and geometric information of an object.\nSecond, we propose to learn a metric that combines the Mahalanobis and feature\ndistances when comparing a track and a new detection in data association. And\nthird, we propose to learn when to initialize a track from an unmatched object\ndetection. Through extensive quantitative and qualitative results, we show that\nwhen using the same object detectors our method outperforms state-of-the-art\napproaches on the NuScenes and KITTI datasets.\n