When applied to the non-linear matter distribution of the universe, neural networks have been shown to be very statistically sensitive probes of cosmological parameters, such as the linear perturbation amplitude $\sigma_8$. However, when used as a"black box", neural networks are not robust to baryonic uncertainty. We propose a robust architecture for constraining primordial non-Gaussianity $f_{NL}$, by training a neural network to locally estimate $\sigma_8$, and correlating these local estimates with the large-scale density field. We apply our method to N-body simulations, and show that $\sigma(f_{NL})$ is 3.5 times better than the constraint obtained from a standard halo-based approach. We show that our method has the same robustness property as large-scale halo bias: baryonic physics can change the normalization of the estimated $f_{NL}$, but cannot change whether $f_{NL}$ is detected.