A 360{\\deg} perception of scene geometry is essential for automated driving,\nnotably for parking and urban driving scenarios. Typically, it is achieved\nusing surround-view fisheye cameras, focusing on the near-field area around the\nvehicle. The majority of current depth estimation approaches focus on employing\njust a single camera, which cannot be straightforwardly generalized to multiple\ncameras. The depth estimation model must be tested on a variety of cameras\nequipped to millions of cars with varying camera geometries. Even within a\nsingle car, intrinsics vary due to manufacturing tolerances. Deep learning\nmodels are sensitive to these changes, and it is practically infeasible to\ntrain and test on each camera variant. As a result, we present novel\ncamera-geometry adaptive multi-scale convolutions which utilize the camera\nparameters as a conditional input, enabling the model to generalize to\npreviously unseen fisheye cameras. Additionally, we improve the distance\nestimation by pairwise and patchwise vector-based self-attention encoder\nnetworks. We evaluate our approach on the Fisheye WoodScape surround-view\ndataset, significantly improving over previous approaches. We also show a\ngeneralization of our approach across different camera viewing angles and\nperform extensive experiments to support our contributions. To enable\ncomparison with other approaches, we evaluate the front camera data on the\nKITTI dataset (pinhole camera images) and achieve state-of-the-art performance\namong self-supervised monocular methods. An overview video with qualitative\nresults is provided at https://youtu.be/bmX0UcU9wtA. Baseline code and dataset\nwill be made public.\n