Learning Transferable Features for Implicit Neural Representations

Summary

This paper introduces STRAINER, a novel training framework for Implicit Neural Representations (INRs). Unlike traditional INRs, which are trained on a single signal, STRAINER aims to learn transferable features that can be effectively utilized for fitting new signals from a similar distribution. The method involves sharing initial encoder layers across multiple INRs while using independent decoder layers. This setup enables STRAINER to achieve faster convergence and better reconstruction quality compared to baseline models. The paper evaluates STRAINER on various tasks, including image fitting, super-resolution, and denoising, demonstrating its effectiveness in both in-domain and out-of-domain scenarios. Detailed analyses are provided to understand the transferability of the learned features and their implications for INR training dynamics.

Strengths

Originality: The paper presents a novel approach to improving the transferability of INRs by sharing initial encoder layers. This is a significant departure from traditional methods that train INRs on a single signal without focusing on feature transferability. Quality: The empirical evaluations are thorough, covering multiple datasets and tasks such as image fitting, super-resolution, and denoising. The experiments convincingly demonstrate the advantages of STRAINER in terms of reconstruction quality and convergence speed. Detailed analysis of training dynamics and feature transferability adds depth to the understanding of STRAINER's performance. Clarity: The paper is well-structured, with clear explanations of the methodology, experimental setup, and results. The inclusion of visualizations and tables enhances the readability and comprehension of the findings. Significance: STRAINER addresses a critical limitation in the current INR landscape by enabling feature transferability, which has broad implications for various applications, including medical imaging and inverse problems. This makes the work highly significant for the field.

Weaknesses

Limited Theoretical Insights: While the empirical results are strong, the paper could benefit from more theoretical insights into why the shared encoder layers facilitate better feature transferability. A deeper theoretical framework could strengthen the contributions. Stability Issues: The paper mentions occasional instability in the form of PSNR drops during test signal fitting. Although STRAINER recovers quickly, addressing this issue more comprehensively would improve the robustness of the method. Comparison with Other Models: The comparison with other advanced models, such as those using CNNs and Transformers for transfer learning, is not fully explored. Including more comparisons could provide a clearer picture of STRAINER's relative performance.

Questions

1.Can the authors provide more theoretical insights or an intuitive explanation for why sharing initial encoder layers results in better feature transferability? 2.How does STRAINER perform when the distribution of training and test signals significantly differs? Are there specific characteristics of the signals that influence the transferability of features? 3.Could the authors elaborate on the stability issues observed during test signal fitting? Are there specific conditions or types of signals where these issues are more pronounced? 4.How does STRAINER compare with other transfer learning techniques that use CNNs or Transformers?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have adequately addressed the limitations of their work, discussing the occasional instability observed during test signal fitting and the need for further characterization of the transferred features.

Summary

This paper proposes STRAINER, where implicit neural representations (INRs) are trained for a class of similar objects, with the INRs for all objects having a shared, instance-agnostic encoder and instance-specific decoders. The paper also examines INR training dynamics. STRAINER is evaluated on 2D image regression and fitting MRI images.

Strengths

Proposes the novel idea of sharing the first $K$ layers of the INR among all INR instances for a class of objects, while having instance-specific decoders for each instance, with some improvements in performance over SIREN/meta-learning baselines. The authors include many experimental details so the experiments should be easy to reproduce. The paper is well-written and easy to understand.

Weaknesses

The paper only compares against one meta-learning baseline, ignoring the large amount of related literature (Trans-INR [1], Instance Pattern Composer (IPC) [2], PONP [3], Locality-aware generalizable implicit neural representation [4], etc.) that also looks at generalizable INRs. In particular, the results of this paper seem incongruent with the results of Instance Pattern Composers, which shows that modulation is most effective in the second layer of an INR (i.e., the second layer should be instance-specific while all other layers should be instance-agnostic). The paper has limited experimental evaluation, limited only to image reconstruction and MRI reconstruction. The paper did not investigate the more complicated tasks that were also investigated by the baselines (e.g. LearnIt), such as novel view synthesis or CT reconstruction. Results on some tasks are also only comparable to the baseline (e.g. superresolution), and some tasks do not include all the baselines (e.g. the Kodak task does not have the LearnIt baseline). No ablation study is done on some key hyperparameters, such as the number of shared encoder layers $N$. In light of other works on generalizable INR, the contribution of this work is limited since it does not compare to many of the other works. References 1. Chen, Yinbo, and Xiaolong Wang. "Transformers as meta-learners for implicit neural representations." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. 2. Kim, Chiheon, et al. "Generalizable implicit neural representations via instance pattern composers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 3. Gu, Jeffrey, Kuan-Chieh Wang, and Serena Yeung. "Generalizable Neural Fields as Partially Observed Neural Processes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. 4. Lee, Doyup, et al. "Locality-aware generalizable implicit neural representation." Advances in Neural Information Processing Systems 36 (2024).

Questions

1. While in a different setting, the IPC paper finds that the optimal instance-specific layer is the second layer of an INR. What accounts for the discrepancy between this paper's findings (the instance-specific layers should be the last few layers) and the findings of the IPC paper? 2. Why is the LearnIt baseline not used for the Kodak task?

Rating

5

Confidence

4

Soundness

1

Presentation

3

Contribution

1

Limitations

Limitations are discussed in the paper. Negative societal impact does not seem to be a concern for this paper.

Summary

The research focuses on learning transferable features using a SIREN model, where encoder and decoder sub-networks are utilized to optimize weights for encoding and decoding. The paper evaluates the main claims made in the abstract and introduction, confirming that the goals are clearly stated and the method effectively addresses them, supported by results. The study discusses fitting unseen signals using optimized encoder weights and highlights the fast convergence of STRAINER to low and high frequencies during training. Experiments involve using the SIREN MLP model with sinusoid nonlinearities, along with different versions of STRAINER compared to baselines, all with equal parameters for fair comparison. Datasets such as CelebA-HQ, AFHQ, and OASIS-MRI are used for experimentation, with the shared encoder of STRAINER trained on 10 images from each dataset for shared initialization in subsequent tests. The research also explores image fitting tasks in-domain and out-of-domain, demonstrating the effectiveness of STRAINER compared to other models, specifically showcasing superior performance in various scenarios like super-resolution and denoising, emphasizing the importance of learned transferable features.

Strengths

- The paper introduces the concept of STRAINER, which demonstrates fast learning of signals at test time by capturing high frequencies efficiently. - It highlights the adaptability of STRAINER's initialization for fitting new signals, showcasing better performance compared to Meta-learned 5K in learning input space partitioning. - Improved in-domain performance with STRAINER compared to meta-learning methods for INRs, providing better input space subdivision that aids in faster convergence on inverse problems like super-resolution and denoising.

Weaknesses

- The paper missed mentioning a couple of important papers on meta-learning INRs, say Functa [A] and Spatial Functa [B]. Especially the latter achieves good PSNR as well as classification accuracy, despite that the methods and the focus are different. [A] E Dupont et al. Your data point is a function and you can treat it like one. [B] M Bauer et al. Scaling Functa to ImageNet Classification and Generation. - The method is only tested on image datasets but not 3D shapes or radiance fields, where INRs are more applied to.

Questions

- Besides the improvement in PSNR, is the generalization ability expected to help with completion of the signal (i.e. inpainting)? - It is mentioned that the encoder trained on CelebA-HQ works even better than in-domain pretraining on AFHQ (cats) and OASIS-MRI datasets. Have you tried out-of-distribution generalization trained on other datasets as source domain?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in Section 5.1.