Simulating interactions between non-spherical colloidal particles is computationally challenging due to the complex dependency of forces and energies on their geometry. Instead of a position and orientation, we represent each shape by a small set of points with the same symmetry, which allows the use of descriptor-based and end-to-end models for predicting interaction energies and forces of atomistic systems. Then, we compare various descriptors coupled with different regression models, such as Behler-Parrinello descriptors, smooth overlap of atomic positions, and neuroevolution potential (NEP), as well as multiple end-to-end models, namely, SchNet, DimeNet, and DimeNet++. Among these, the NEP offers an optimal balance between accuracy and computational efficiency. NEP, originally developed for atomistic systems, represents interactions between rigid anisotropic bodies using point clouds, which enables the representation of any arbitrary shape. Molecular dynamics simulations using NEP accurately reproduced structural properties across diverse particle shapes, including cubes, tetrahedra, pentagonal bipyramids, and twisted cylinders, while achieving roughly an order-of-magnitude speedup over other methods. In addition, we show that the extension of the method to multi-face shapes with different interactions on their surface is straightforward. We used a twisted cylinder, which lacked any point group symmetry, to demonstrate the flexibility and accuracy of NEP. Our approach enables scalable simulations of complex colloidal systems and can potentially help to facilitate efficient studies on shape dependent interactions and phase behavior in the future.