Accurate forecasting of renewable generation is crucial to facilitate the integration of Renewable Energy Sources (RESs) into the power system. Focusing on photovoltaic (PV) units, forecasting methods can be divided into two main categories: physics-based and data-based strategies, with Artificial Intelligence (AI)-based models providing state-of-the-art performance. However, while these AI-based models can capture complex patterns and relationships in the data, they rarely exploit the physics-derived information produced by Numerical Weather Prediction (NWP) models. Therefore, in this paper, we propose MATNet, a novel transformer-based multimodal architecture for multi-step day-ahead PV power generation forecasting. The model is fed with historical PV data and historical and future weather covariates through a multi-level joint fusion approach, employing a soft-attention mechanism at multiple fusion stages. We evaluate MATNet on the Ausgrid benchmark dataset. MATNet achieves an RMSE of 0.0445, corresponding to a relative improvement of approximately 65% over the best-performing external baseline. The evaluation includes ablation studies, a sensitivity analysis on missing data, a cross-site zero-shot evaluation on five external PV datasets, and a computational complexity analysis. These experiments assess the contribution of each input modality, the resilience to input degradation, the transferability across PV sites, and the trade-off between accuracy and computational cost. These results highlight MATNet's potential as a reliable and efficient solution to facilitate the integration of PV energy into the power grid. The code is available at https://github.com/arco-group/MATNet .