Activity detection from first-person videos (FPV) captured using a wearable\ncamera is an active research field with potential applications in many sectors,\nincluding healthcare, law enforcement, and rehabilitation. State-of-the-art\nmethods use optical flow-based hybrid techniques that rely on features derived\nfrom the motion of objects from consecutive frames. In this work, we developed\na two-stream network, the \\emph{SegCodeNet}, that uses a network branch\ncontaining video-streams with color-coded semantic segmentation masks of\nrelevant objects in addition to the original RGB video-stream. We also include\na stream-wise attention gating that prioritizes between the two streams and a\nframe-wise attention module that prioritizes the video frames that contain\nrelevant features. Experiments are conducted on an FPV dataset containing $18$\nactivity classes in office environments. In comparison to a single-stream\nnetwork, the proposed two-stream method achieves an absolute improvement of\n$14.366\\%$ and $10.324\\%$ for averaged F1 score and accuracy, respectively,\nwhen average results are compared for three different frame sizes\n$224\\times224$, $112\\times112$, and $64\\times64$. The proposed method provides\nsignificant performance gains for lower-resolution images with absolute\nimprovements of $17\\%$ and $26\\%$ in F1 score for input dimensions of\n$112\\times112$ and $64\\times64$, respectively. The best performance is achieved\nfor a frame size of $224\\times224$ yielding an F1 score and accuracy of\n$90.176\\%$ and $90.799\\%$ which outperforms the state-of-the-art Inflated 3D\nConvNet (I3D) \\cite{carreira2017quo} method by an absolute margin of $4.529\\%$\nand $2.419\\%$, respectively.\n