NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder

Interpolation for scattered data is a classical problem in numerical analysis, with a long history of theoretical and practical contributions. Recent advances have utilized deep neural networks to construct interpolators, exhibiting excellent and generalizable performance. However, they still fall short in two aspects: <bold>1) inadequate representation learning</bold>, resulting from separate embeddings of observed and target points in popular encoder-decoder frameworks and <bold>2) limited generalization power</bold>, caused by overlooking prior interpolation knowledge shared across different domains. To overcome these limitations, we present a <bold>N</bold>umerical <bold>I</bold>nterpolation approach using <bold>E</bold>ncoder <bold>R</bold>epresentation of <bold>T</bold>ransformers (called <bold>NIERT</bold>). On one hand, NIERT utilizes an encoder-only framework rather than the encoder-decoder structure. This way, NIERT can embed observed and target points into a unified encoder representation space, thus effectively exploiting the correlations among them and obtaining more precise representations. On the other hand, we propose to pre-train NIERT on large-scale synthetic mathematical functions to acquire prior interpolation knowledge, and transfer it to multiple interpolation domains with consistent performance gain. On both synthetic and real-world datasets, NIERT outperforms the existing approaches by a large margin, i.e., 4.3<inline-formula><tex-math notation="LaTeX">$\sim 14.3\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mn>14</mml:mn><mml:mo>.</mml:mo><mml:mn>3</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="ding-ieq1-3402444.gif"/></alternatives></inline-formula> lower MAE on TFRD subsets, and 1.7/1.8/8.7× lower MSE on Mathit/PhysioNet/PTV datasets.