Curvature in form of the Hessian or its generalized Gauss-Newton (GGN)\napproximation is valuable for algorithms that rely on a local model for the\nloss to train, compress, or explain deep networks. Existing methods based on\nimplicit multiplication via automatic differentiation or Kronecker-factored\nblock diagonal approximations do not consider noise in the mini-batch. We\npresent ViViT, a curvature model that leverages the GGN's low-rank structure\nwithout further approximations. It allows for efficient computation of\neigenvalues, eigenvectors, as well as per-sample first- and second-order\ndirectional derivatives. The representation is computed in parallel with\ngradients in one backward pass and offers a fine-grained cost-accuracy\ntrade-off, which allows it to scale. We demonstrate this by conducting\nperformance benchmarks and substantiate ViViT's usefulness by studying the\nimpact of noise on the GGN's structural properties during neural network\ntraining.\n