3D semantic segmentation is a fundamental building block for several scene\nunderstanding applications such as autonomous driving, robotics and AR/VR.\nSeveral state-of-the-art semantic segmentation models suffer from the part\nmisclassification problem, wherein parts of the same object are labelled\nincorrectly. Previous methods have utilized hierarchical, iterative methods to\nfuse semantic and instance information, but they lack learnability in context\nfusion, and are computationally complex and heuristic driven. This paper\npresents Segment-Fusion, a novel attention-based method for hierarchical fusion\nof semantic and instance information to address the part misclassifications.\nThe presented method includes a graph segmentation algorithm for grouping\npoints into segments that pools point-wise features into segment-wise features,\na learnable attention-based network to fuse these segments based on their\nsemantic and instance features, and followed by a simple yet effective\nconnected component labelling algorithm to convert segment features to instance\nlabels. Segment-Fusion can be flexibly employed with any network architecture\nfor semantic/instance segmentation. It improves the qualitative and\nquantitative performance of several semantic segmentation backbones by upto 5%\nwhen evaluated on the ScanNet and S3DIS datasets.\n