LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes

Neural Radiance Fields (NeRFs) have shown remarkable performances in producing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultaneously suffer from low visibility, noise, and camera shakes. While existing NeRF methods may handle either low light or motion, directly combining them or incorporating additional image-based enhancement methods does not work as these degradation factors are highly coupled. We observe that noise in low-light images is always sharp regardless of camera shakes, which implies an implicit order of these degradation factors within the image formation process. To this end, we propose in this paper a novel model, named LuSh-NeRF, which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images. The key idea of LuSh-NeRF is to sequentially model noise and blur in the images via multi-view feature consistency and frequency information of NeRF, respectively. Specifically, LuSh-NeRF includes a novel Scene-Noise Decomposition (SND) module for decoupling the noise from the scene representation and a novel Camera Trajectory Prediction (CTP) module for the estimation of camera motions based on low-frequency scene information. To facilitate training and evaluations, we construct a new dataset containing both synthetic and real images. Experiments show that LuSh-NeRF outperforms existing approaches. Our code and dataset can be found here: https://github.com/quzefan/LuSh-NeRF.

Paper

Similar papers

Peer review

Reviewer m3Lt6/10 · confidence 4/52024-07-07

Summary

This work proposes a model to reconstruct a clean and sharp NeRF from a set of hand-held low-light images. The authors recognize the implicit order of the degradations (blur, noise, and low visibility), and sequentially decouple and remove each degradation in the network training. An SND module is proposed for noise removal and a CTP module is introduced for deblurring. They have conducted experiments on a dataset constructed from the LOL-Blur dataset proposed by [61].

Strengths

1. This is the first work to reconstruct degradation-free images from a set of low-light photographs with motion blur. 2. They have introduced a new method for scene-noise decomposition from the implicit scene representation and utilized the frequency information for accurate kernel prediction NeRF. 3. The qualitative results are appealing and the authors have shown better quantitative scores for the method.

Weaknesses

1. The presentation of the problem through mathematical equations is not clear. In Eq. 5, $C_{noisy}$ is the color value of a bright light blurry noisy (given in Eq. 4) image. But in Eq. 5, the first term $C_{S-NeRF}$ output is the sharp image and the second term is the noise term. The blurring operation is missing in this equation. Please correct me if I misinterpret anything. Also, N-NeRF does not use any volume rendering calculation to find the noise pixel value. Then, why this module is called a NeRF? It simply finds a pixel value for a given input pose. How the second term in Eq. 5 ($n\frac{N}{2}$) is derived? In Eq. 6, what are $i$ and $j$? In Fig. 2, the align ray is given subscripts $j$ and $k$. Please be consistent with equations and figure. 2. In the paper, it is given that the camera trajectory prediction (CTP) module is novel (Line 16, 47). But the same idea of [19] (Line 180) is used for camera trajectory prediction with the additional masking of high-frequency pixel regions in the image. The main function of deblur-NeRF in the CTP module is not novel. 3. Edge portions in the image will have high frequencies. When only rays containing low frequencies are used for gradient computation, typically edges (or high-frequency regions) will not come into the picture. But for deblur-NeRF, excluding the edge areas (high-frequency) is not a good idea since the blurring is mostly visible in the edges other than homogeneous (low-frequency) areas. How your frequency filter radius is selected? Did you consider any such factors for its selection? 4. In Line 224, it is given that for the first 60K iterations, $\beta$ is 0. This means SND does not come into the picture. In that case, the blur kernel estimated (by deblur-NeRF) will be wrong right (according to lines 185-186)? Is it correct to start from these trained weights after 60K iterations? What happens if $\beta$ is not 0 in the initial iterations? 5. SND relies on the image-matching method [40]. How generalizable is the method to your dataset? Have you taken the pre-trained weights? 6. The contribution of the dataset (given in Line 64) is not that significant since the poses of the available dataset from [61] are computed for the dataset generation using COLMAP. The method is tested on this single dataset (containing synthetic and real videos). Since the work is meant for handheld lowlight photography, a video of a low-light scene with handheld cameras (which typically will have a camera shake) could have been captured and the proposed method could have been tested on the dataset. 7. You have missed several references for Deblur NeRF papers. "ExBluRF: Efficient Radiance Fields for Extreme Motion Blurred Images", ICCV 2023; "PDRF: progressively deblurring radiance field for fast scene reconstruction from blurry images", AAAI 2023; "Inverting the Imaging Process by Learning an Implicit Camera Model", CVPR 2023. The related works (Sec. 2) section is not proper. There should be subsections for deblur NeRF and lowlight NeRF and briefly explain each work in both, other than explaining NeRF. Also, only low-light image enhancement methods are included. Why are deblurring methods not included? Or, it is better to have a subsection with explanations of lowlight deblurring works [61, 56]. Instead of just giving reference numbers, the essence of each very relevant work should be explained in a sentence in 'Related works'. 8. Some typos are there. In Line 153 and Fig. 6 (a) caption. In Line 35, [29] is not for low-light scenes. Line 41, the abbreviation of ISP is missing.

Questions

Please see weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes. Limitations are included in Section 5 and negative societal impacts are included in Section A.1.

Reviewer 4eod4/10 · confidence 3/52024-07-10

Summary

The paper proposed a method to train a NeRF with blurry (due to camera motion), low-light scene images. After training the method allows the recovery of enhanced, sharp images. To solve the problem two modules are proposed: 1) A SND module for noise modeling, which includes both a noise-prediction NeRF (N-NeRF) and a consistency loss, which uses image matching to constrain the radiance to be multiview consistent 2) A CTP module to model the blur induced by camera motion The paper evaluates the method and baselines on a self-collected dataset.

Strengths

Originality: I appreciate the problem being tackled in the paper and think this is an underexplored problem. I think we are still far from being able to use NeRFs on casually captured phone videos, as evidenced by apps such as Luma, which although produce excellent quality require no (camera) motion blur and perfect lighting in the captures. I also appreciate the authors trying to assemble their own dataset although I do have some qualms about their claims (see weakness section). Quality: I think the experiments and baselines have been chosen well.

Weaknesses

Quality: Even though I believe the dataset to be useful, I find it hard to attribute the dataset to the authors of the paper. If I understand correctly (please correct me if I am wrong), the original images were actually captured by prior work (Lednet), and the authors of LuSh-NeRF only ran an off-the-shelf SFM method to recover the camera parameters? Happy to be shown otherwise, but I do not think in this case the authors can claim the dataset as a contribution of the paper. Other than that although the results seem convincing, the contribution of the paper isn’t significant. Clarity: I had serious issues understanding the paper, specifically the methods section. Authors introduce a lot of names, which I think would be better replaced by just mathematical notation. Some of the notation is also not properly explained, for example, $n_{N/2}$ in Eq. 5. (see also questions). It is also unclear to me how exactly Equations 4 and 5 relate. If I understand correctly Eq 4. suggests that to render the low-light, blurry images the network prediction is first unsharpened and then noise is added, but in Eq 5. noise is directly added to the radiance predictions from the network? I think part of the CTP module also comes from the paper DP-NeRF, but I think the paper does not exactly mention what this module does. In the interest of making the paper self-sufficient, I think the authors should elaborate on what the module does/what it is exactly. I think what they do elaborate on (the detaching of certain rays) is just their contribution on top of the method from DP-NeRF, if I understand correctly. A few typos, there’s probably more I missed: L145: bracket not closed L183: bracket not opened

Questions

1) I think the $n_{N/2}$ refers to the N-NeRF output at the 3D coordinate which is the midpoint of the ray samples, is that right? If we don’t care about the 3D consistency of N-NeRF (since noise is not 3D consistent) why is the noise even modeled with a NeRF? Why not just optimize a tensor per view, or have the noise be a ray-conditioned network (instead of 3D coordinate conditioned)? 2) I wonder how important it is to low-pass filter the images before thresholding for the CTP module, have the authors tried to just threshold the image without the low-pass filter?

Rating

4

Confidence

3

Soundness

3

Presentation

1

Contribution

2

Limitations

Yes.

Reviewer 4eod2024-08-11

Thank you for the response to my comments. **W1: Insufficient Dataset Contribution.** For the discussion on the dataset, I am still not satisfied, I would not attribute the paper any significant contribution for selecting views for 5 synthetic and 5 captured scenes from an existing dataset and estimating the camera parameters, even though I appreciate COLMAP is not always straightforward to use. **W4: Relation between CTP module and DP-NeRF.** Thank you very much for this explanation, that helps me better understand the difference between the two modules. But frankly, I think even this explanation is not enough for what the RBK module exactly does, I think its common practice to explain properly the methods used if they are non-standard even if they come from other papers. I would implore the authors to add a more thorough description, even if just in the supplement. **Q1. Why is the noise modeled with a NeRF?** This begs the question, why the authors don't use the (x, y, N) based formulation. Isn't this formulation more efficient and understandable? Presumably, this MLP is smaller, and since there is no spatial consistency required, it makes more sense to use it than a 3D-conditioned MLP? EDIT: Maybe I am misunderstanding something, what is the ray midpoint the author's query? This changes as per the random sampling, is that right? So for the same pixel, I could be querying a different midpoint in different iterations? Otherwise, I thank the authors for their responses, I am satisfied with the answers provided to all other questions. I think the discussion on the low-pass filter is especially useful, since the Freq. Domain Thresholding is one of the main contributions to the CTP module. I would implore the authors to add these results to the paper.

Authorsrebuttal2024-08-12

Thanks for your reply and we are glad to see that our response can address most of your raised concerns. We would like to futher clarify the below issues. ### R3W1: Insufficient Dataset Contribution. **R:** (1) The dataset contribution is one **part of** our **3rd contribution** and we still have **other technical contributions**. (2) Note that our dataset is necessary as we are handling a **new task** (handling NeRF in low-light scenes with camera motions). To construct this dataset, it took us more than 3 full weeks to select images and tune the camera pose parameters. Note that the colmap-free method [A] does not handle our task well as it is **very difficult** to optimize the camera pose directly in **low-light blurry** scenes. (3) We can **revise** our third contribution to **emphasize more** on the experimental evaluations and state-of-the-art results of our model. However, we **do need this dataset** for evaluation and we did put efforts into constructing this dataset. [A] NeRF--: Neural radiance fields without known camera parameters, arXiv:2102.07064, 2021. ### R3W4: Relation between CTP module and DP-NeRF. **R:** Thansk for your suggestion. The Rigid Blurring Kernel (RBK) module in DP-NeRF[19] models the scene blur kernel by simulating the 3D camera motions via the following two main parts: **Ray Rigid Transformation (RRT):** The RRT models the blurring process of an image. It is formulated as ray transformation derived from the deformation of rigid camera motion, which is defined as the dense SE(3) field for scene $s$ and modeled by the MLPs as: $S_s = (r_s;v_s) = (\mathcal{R}(\mathcal{E}(l_s);\mathcal{L}(\mathcal{E}(l_s))), where \ s \in view_{img},$ where $l_s$ is the latent code for each view through the embedding layer in [B], $\mathcal{R}, \mathcal{L}, \mathcal{E}$ are three MLP networks, $view_{img}$ is the training view set. The $S_s = (r_s;v_s) \in \mathbb{R}^6$ is the matrix which will be used for the RRT modeling as follows: $ray_{s;q}^{RRT} = Rigid-Transform(ray_s, (r_s;v_s)),$ where $Rigid-Transform()$ function is the standard 3D rigid transformation operation, $ray_s$ and $ray_{s;q}^{RRT}$ are the orgin ray and the transformed rays in scene $s$, $q \in \\{1,...,k \\}$, $k$ is a hyper-parameter that controls the number of camera motions contributing to the blur in each scene $s$. The blurry RGB value at $ray_s$ can be acquired by weighting sums of the NeRF volume rendering values $C_{s;0}$ and $C_{s;q}$ from $ray_s$ and $ray_{s;q}^{RRT}$. **Coarse Composition Weights:** The Coarse Composition Weights are computed for each ray obtained by the RRT: $m_{s;0,1,...,k} = \sigma(\mathcal{W}(\mathcal{E}(l_s))), where \ \sum_{i=0}^{k}m_{s;i} = 1,$ where $m_s$ is the final weight for each ray in RRT. Finally, blurry color $\mathnormal{C}_s$ for scene $s$ can be computed by the weighted sum operation as shown below: $C_s = m_{s;0}C_{s;0} + \sum_{q=1}^{k}m_{s;q}C_{s;q}.$ **We will add these information to the revision.** [B] Optimizing the latent space of generative networks, arXiv:1707.05776, 2017. ### R3Q1: Why is the noise modeled with a NeRF? Why not optimize ... **R:** We agree with the reviewer that the $(x,y,N)$-based formulation is another possbile implementation of modeling noise comparing to our current one (the rendering results of both **do not differ much**). Note that this **does not affect our goal** of decomposing the scene and noise information. Regarding the ray midpoint, we **uniformly sample (instead of random sampling in S-NeRF)** between the near and far fields (calculated by COLMAP) of the ray, and select the coordinates $P_{mid}$ of the intermediate sampling point, along with the view direction $d$, as the input of N-NeRF. Since the **bounds of the ray do not change** during the training phase, the midpoints for each ray (one-to-one map with pixels) is same. We will incorporate all these information into our revision.

Authorsrebuttal2024-08-13

Dear Reviewer 4eod, Thank you again for your review and reply. We hope that our rebuttal and the following comments could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper. Best regards, Authors of Paper #10302

Reviewer 4eod2024-08-14

Thank you very much! I appreciate that the discussion is coming to an end, I don't think I need any more input from the authors. I'm not sure I will increase my score but I will relay my thoughts to the AC. **Method section.** Ultimately, I would just like to implore the authors to rewrite the methods section a bit more clearly. My suggestion would be to limit the use of these short forms such as RRT, CTP etc. I think it's difficult to keep track of all these. Maybe use just two short forms for the two main parts of the method (SND, CTP). And make separate paragraphs within sections to describe each component of these separate paths. **Dataset.** I would also de-emphasize the dataset claim a bit, as the authors suggested. Thanks a lot!

Authorsrebuttal2024-08-14

Thanks for your suggestions. We promise to **revise our Method section** in the revision to ensure better understanding.

Reviewer 54cf6/10 · confidence 5/52024-07-12

Summary

The authors propose LuSh-NeRF, a model that reconstructs a clean and sharp NeRF from handheld low-light images by sequentially modeling noise and blur. LuSh-NeRF includes a Scene-Noise Decomposition (SND) module for noise removal and a Camera Trajectory Prediction (CTP) module for estimating camera motions based on low-frequency scene information. Experiments demonstrate that LuSh-NeRF outperforms existing methods in rendering bright and sharp novel-view images from low-light scenes.

Strengths

1.The paper aims to address the challenge of using NeRFs in low-light conditions, where images often suffer from low visibility, noise, and camera shakes together. 2. A new dataset containing synthetic and real images is constructed to facilitate training and evaluation. 3. The method is grounded in a sound theoretical framework, leveraging multi-view consistency and frequency-domain information.

Weaknesses

1. For me, the ideas and motivation behind this paper are quite good. What I am concern is whether the authors could consider adding some ablation studies, such as the role of various modules in LuSh-NeRF, this could let reader know which part is more effective. 2. In the scenario described by the author, I think the colmap-free NeRF method of this task may be more effective. Low-light blurred images have a certain impact on colmap estimation.

Questions

1. I have another small suggestion. Could the authors consider compressing the images, or perhaps saving them in PDF format? This might help reduce the memory size of the paper's PDF file. 2. Could the author show some comparison with Aleth-NeRF [7] ? which is also a low-light NeRF method.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Please refer to the weakness part.

Reviewer 4gW76/10 · confidence 4/52024-07-13

Summary

This method proposes a solution for NeRF optimization under low light settings by resolving 3 different forms of degradation: low intensity, camera noise, and motion blur. Low intensity is effectively resolved by scaling up the image, camera noise is resolved by proposing a consistency loss between different views to locate the noise, and motion blur is handled by using a camera trajectory prediction module to predict the camera motion and thus the corresponding sharpening function. The experiments demonstrate state-of-the-art performance on novel view synthesis given low-light conditions. Qualitative ablations are provided to support the contribution of each component of the method. The authors will also make their code readily available as well as a dataset with 5 synthetic and 5 real scenes for evaluating low-light novel view synthesis.

Strengths

The experiments support that the model is state-of-the-art, as in most scenes the proposed method achieves the best performance for low-light novel view synthesis. The model is well thought out and handles many types of degradations, including low intensity, noise, and motion blur. The SND and CTP modules are well designed and novel components of the method. The paper presentation is clear and easy to follow with appropriate figures to aid the reader's understanding. The code will be made publicly available and a dataset will be released to aid evaluations in the low-light novel view synthesis domain.

Weaknesses

The ablations would be more convincing with some quantitative results to back up the qualitative results. There appear to be some mistakes in Table 1: in some columns two numbers are underlined, in others the best and second best seem to be incorrectly reported. Please be careful to fix these mistakes. There are some important citations missing for some parts of the related work, such as in line 71 when mentioning NeRF methods that model digital humans: 1. HeadNeRF (CVPR 2022) 2. MoFaNeRF (ECCV 2022) 3. INFAMOUS-NeRF (arxiv, 2023)

Questions

As mentioned above, please provide quantitative results to support the ablation studies and be sure to fix the errors in Table 1. I will be considering the rebuttal carefully as well as the thoughts from other reviewers in deciding my final rating.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations and broader impact section are both available and appropriate. This work naturally does not have many immediately obvious negative societal impacts as it is simply trying to faithfully render novel-view images in low-light conditions.

Reviewer m3Lt2024-08-09

Clarification on W1Part1 response

Thank you for your response. Most of my concerns are addressed by the authors. But I have one small doubt in the response. In the updated Eq. 5 given in W1Part1 response, $MLP_{N-NeRF}(P_{mid}, d) = n N/2$. But Line 159 of the paper says that '$n$ is the noise value rendered by N-NeRF'. Which one is correct? $MLP_{N-NeRF}(P_{mid}, d) = n N/2$ or $MLP_{N-NeRF}(P_{mid}, d) = n$

Authorsrebuttal2024-08-09

Clarification on W1Part1

Thanks for the reply. We are glad that our rebuttal could address your concerns. $n_{\frac{N}{2}} = MLP_{N-NeRF}(P_{mid}, d)$ is correct. We will correct $n$ into $n_{\frac{N}{2}}$ in Line 159 and check elsewhere to maintain the consistency of symbols in our revision.

Reviewer m3Lt2024-08-09

Thank you for the clarification. I am increasing the rating to 'weak accept'.

Authorsrebuttal2024-08-10

Thanks very much for your scores and review comments, your detailed comments effectively help us to improve the quality of manuscript. We promise to incorporate all these missing details in the revision.

Reviewer 4gW72024-08-09

Thanks for your thorough rebuttal! Please include these missing details in the final version (especially the ablation results) to help deliver a more convincing set of experiments. I will maintain my Weak Accept rating as I see most other reviewers are in agreement.

Authorsrebuttal2024-08-10

Thanks a lot for your acknowledgement and valuable comments, we promise to incorporate all these missing details in the revision.

Authorsrebuttal2024-08-10

We're very glad that our rebuttal has addressed your concerns, and thank you for recognizing the paper, we'll add the missing details in the revision.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC