We present a novel, real time algorithm to compute the trajectory of each pedestrian in moderately dense crowd scenes. Our formulation is based on an adaptive particle filtering scheme that uses a multi-agent motion model based on velocity-obstacles, and takes into account local interactions as well as physical and personal constraints of each pedestrian. Our method dynamically changes the number of particles allocated to each pedestrian based on different confidence metrics. Additionally, we use a new high-definition crowd video dataset to evaluate the performance of different pedestrian tracking algorithms. This dataset consists of videos of indoor and outdoor scenes recorded at different locations, each with 30-80 pedestrians. Using this dataset, we highlight the performance benefits of our algorithm over prior techniques. In practice, our algorithm can compute trajectories of tens of pedestrians on a multi-core desktop CPU at interactive rates (27-30 frames per second). To the best of our knowledge, our approach is 4-5 times faster than prior methods that provide similar accuracy.