NCDL: A Framework for Deep Learning on non-Cartesian Lattices

Summary

This paper generalizes common machine learning operations from Cartesian lattices to other regular lattices, such as the hexagonal lattice. The authors argue that the Cartesian lattice is a sub-optimal representation for important natural signals, and that operating on their non-Cartesian structure natively leads to more efficient implementations and better results. The authors further promise to release a software library for non-Cartesian deep learning, and include an experimental section with implementation details and efficiency arguments, as well as experimental results on various computer vision tasks.

Strengths

The contribution is very novel in the sense that the field of deep learning on lattices is under-explored. The author's work is likely to have a big impact on this area of machine learning, as providing a complete optimized library with non-Cartesian version of common lattice operations with speed-up future research significantly.

Weaknesses

The paper is short, with the authors being allowed one more page of content. They could improve the discussion by expanding, for example, on the computation of the derivatives. Are the backward passes of all operations naturally handled by PyTorch or did it require manual implementations? What about numerical stability?

Questions

The area of non-Cartesian deep learning could reasonably be considered a sub-field of geometric deep learning, which also includes deep learning on graphs, and models that leverage group symmetry. The authors should consider comparing lattice neural network approaches with GNNs (similar to experiments using sub-pixel graphs), as the graph models should be able to handle the non-Cartesian grid (albeit differently and with less inductive bias).

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Some aspects are lacking, as explained above: - Derivatives and numerical stability - Comparison with other non-Cartesian models such as GNNs and group equivariant neural networks

Summary

This works introduces a framework as well as software for computing convolutions on non-Cartesian lattices. The method is compared to existing software for hexagonal lattices as well as on image data.

Strengths

The method is put in a strong theoretical framework that also explores the very important up and down sampling operations on non-uniform grids. The authors also make available open-source software that is more general and whose performance seems much better than anything available.

Weaknesses

In the context of scientific machine learning, there have been similar ideas explored on how to make architectures which work on arbitrary grids, for example, https://arxiv.org/abs/2207.05209, https://arxiv.org/abs/2305.19663, and I am sure there are others. Furthermore, graph neural networks can also perform convolutions or arbitrary grids, for example, https://arxiv.org/abs/1704.01212, among many, many others. It would be good to mention some of these and even include some comparisons.

Questions

The motivation for images on hexagonal grids are is a bit unclear to me. I always think of images as living on Cartesian grids, so it is somewhat strange to consider them on hexagonal grids instead. Why is this useful? Furthermore, where do you see most of the applications for this? Would be great to include possible large scale applications that lead a vision for future work.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed limitations and potential negative societal impact.

Summary

This paper introduces a high-quality software extension for PyTorch that enables seamless computations with non-Cartesian lattices for 1D, 2D and 3D images. The key observation made by the authors of this work is that **non-Cartesian lattices can be decomposed as "sums" of Cartesian lattices**: up to some clever refactoring (which makes up the core numerical code of the proposed software package), efficient computations on non-Cartesian lattices can directly leverage the standard (and highly optimized) PyTorch/cuDNN implementations of e.g. convolutional layers. This implies that **the proposed software package is both efficient and easy to maintain in the long run**.

Strengths

- The authors tackle an interesting and **very original topic**. Non-Cartesian lattices are indeed fundamental to low-level image processing but essentially absent from the machine learning literature. - The paper is **extremely well written**, with clear figures, attention paid to details and a satisfying evaluation. I haven't tried to run the code provided in the supplementary materials (too many papers to review at once!), but this is **clearly a high-quality software package** with a clean structure, a full test suite and extensive documentation. - This package targets successfully one specific and interesting operation in image processing, providing a **useful extension to our common toolbox via a neat and clever software package**. This is more than what most papers (never mind submissions) can provide and, in my opinion, clearly warrants publication at NeurIPS.

Weaknesses

Reviewers can always ask for more (outstanding deep learning experiments, better run times, support for all types of attention layers and niche hardware architectures, etc.)... But realistically, I am very happy with the paper as submitted.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Summary

he authors claim that the concept of tensors, a fundamental cornerstone in machine learning, assumes data are organized on Cartesian grids. They further suggest that alternative non-Cartesian representations may be more beneficial in certain situations. One case is when the data is inherently non-Cartesian — for example, the raw R/G/B image data from most imaging sensors are follows a quincunx (checkerboard) structure in the green channel. Another case is when non-Cartesian grids show superior performance in certain aspects — for example, the hexagonal lattice is known as the optimal sampling lattice in 2D. The authors hereby propose a framework and a software library that introduces standard machine learning operations on non-Cartesian data. The new data structure is called lattice tensor and the software library is called Non-Cartesian Deep Learning (NCDL).

Strengths

1. In the introduction section, the authors provided sufficient context on why non-Cartesian grid structures may be superior to Cartesian counterparts under certain circumstances. 2. We shall appreciate the effort in designing the memory-efficient coset representation for operations on the hexagonal lattice (shown in Figure 2).

Weaknesses

While custom tensor definitions and operations for non-Cartesian data representation are theoretically pleasing, the authors have not shown clearly the potential applications and impact. It is not obvious which datasets and/or standard machine learning tasks will directly benefit from the non-Cartesian representation. A table detailing some typical use cases will be utterly helpful. The experiments/comparisons performed are not convincing enough. The two main results shown in the submission are (1) runtime of convolution operation at different grid sizes, and (2) loss curves of a Cartesian vs. Quincunx auto-encoder on CelebA (celebrity faces) dataset. In the first experiment, while the authors compare the runtime against the standard Cartesian Conv2D and another non-Cartesian baseline (HexagDLy), they have only investigated the convolution operation but skipped the other operations such as pooling, downsampling, upsampling, gradient computation, and back-propagation. Further, no comment has been made on numerical correctness or precision. In the second experiment, the authors aim to show superior performance of Quincunx auto-encoder for image reconstruction. It is a bit weak as the experiment is only performed on one task over one dataset. It may be helpful to include a few more datasets — they don’t even need to be big one, e.g., STL-10, SVHN, LSUN will be sufficient. Besides, the only metrics shown are L1 and SSIM on the validation set. I would recommend including other metrics such as L2, PSNR, FID, and perceptual distance.

Questions

1. Continuing on Weakness #1, it will be very helpful if the authors can construct a table which outlines several typical use cases, with the following information: (1) domain (vision/language/graph/etc), (2) dataset name and description of data format, (3) machine learning task (classification/segmentation/reconstruction/retrieval/etc), (4) preferable non-Cartesian representation, (5) short explanation on why it’s better than Cartesian. Do you think it is a reasonable idea, or is there a better way to demonstrate the applications and impact? 2. Open-ended question: will it be better to embed the proposed method in hardware?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

While it has been covered in previous sections, the main limitations are: -Unclear potential application and impact. -Insufficient of experiments and comparisons.

Acknowledgement of the rebuttal

You are welcome. I am very satisfied with the paper: having read all reviews and rebuttals, I am convinced that this submission is a very clear accept. Good luck!