Use of physics-based simulation as a planning model enables a planner to\nreason and generate plans that involve non-trivial interactions with the world.\nFor example, grasping a milk container out of a cluttered refrigerator may\ninvolve moving a robot manipulator in between other objects, pushing away the\nones that are movable and avoiding interactions with certain fragile\ncontainers. A physics-based simulator allows a planner to reason about the\neffects of interactions with these objects and to generate a plan that grasps\nthe milk container successfully. The use of physics-based simulation for\nplanning however is underutilized. One of the reasons for it being that\nphysics-based simulations are typically way too slow for being used within a\nplanning loop that typically requires tens of thousands of actions to be\nevaluated within a matter of a second or two. In this work, we develop a\nplanning algorithm that tries to address this challenge. In particular, it\nbuilds on the observation that only a small number of actions actually need to\nbe simulated using physics, and the remaining set of actions, such as moving an\narm around obstacles, can be evaluated using a much simpler internal planning\nmodel, e.g., a simple collision-checking model. Motivated by this, we develop\nan algorithm called Planning with Selective Physics-based Simulation that\nautomatically discovers what should be simulated with physics and what can\nutilize an internal planning model for pick-and-place tasks.\n