LaSe-E2V: Towards Language-guided Semantic-Aware Event-to-Video Reconstruction

Author response to Reviewer H8fU (1/2)

Thank the reviewer for elaborating on the points. We address the reviewer's concerns below: > **Hallucinations** > - Hallucination is a potential **side effect** introduced by incorporating semantic priors from the diffusion model into our framework. In this regard, our proposed techniques are demonstrated to be effective in reducing hallucinations, but **not entirely** eliminating them, which is still a challenge in the diffusion model research community. Extensive experimental results (Tab. 1 and Fig. 4) show that our method significantly **improves fidelity** and reduces hallucinations in the scenes with sufficient event data. Our method also exhibits better reconstruction performance for the scenes even with insufficient event data (Tab. A-6). Please be reminded that it is impossible to totally remove the hallucination effect for regions with insufficient events. However, our method indeed effectively leverages the semantic prior in the diffusion model to reconstruct the image more **closer to the distribution** to the real scene, whereas the previous E2V methods (e.g., HyperE2VID) **fail** in these regions, as shown Fig. 1. - Our framework mainly focuses on providing a potential pipeline to reconstruct the video from events with the guidance of language. This is based on our finding that language naturally conveys abundant semantic information, which is beneficial in enhancing the **semantic consistency** for the reconstructed video (see Lines 39-41). The text guidance can serve as a form of human intervention through text prompts, which is **similar** to the way of the text-guided denoising [80,81] and super-resolution[82] methods. - As shown in Fig. 1, it compares the difference between our method and the previous E2V method (i.e., HyperE2VID). For region with insufficient events, although our results still differs from reality in some details, they indeed reconstructs a scene according to human preference. In contrast, the previous E2V method always reconstructs **haze** in these regions. For Fig. 7, we will update the results from previous E2V methods for clearer comparison. Quantitatively, Tab. A-6 also shows the superiority of our method in the scene with insufficient events. > **Ambiguity in Text Prompts** > - Our framework incorporates text descriptions as complementary information, marking the first instance of allowing language guidance in the E2V pipeline. This approach is believed to "*inspire future work*" (***ueNs***) and "*provide new ideas*" (***uWG8***). While this may introduce some ambiguity, it also offers the flexibility to manually adapt the reconstructed video according to user preferences, as demonstrated in Fig. 8. The text guidance manner is also similar to the way of the text-guided denoising [80,81] and super-resolution[82] methods. - In regions with sufficient event data, our method improves reconstruction performance by leveraging the semantic priors from the text prompts, ensuring high fidelity. For regions with insufficient event data, the method relies solely on the text prompts to reconstruct a scene that aligns with human preferences. Although this approach may produce textures or details that differ from the real ones, it still generates images that are closer to reality. In contrast, the previous E2V models always reconstruct these regions as indistinct haze, far from the true distribution of real images. Fig. 1 shows the qualitative comparison. Tab. A-6 also shows the quantitative comparison and demonstrates the superiority of our method. We will add more comparisons on these scenes with insufficient events in the revision. > **Computational Resources and Inference Times** > - Table A-5 provides the GPU memory for different E2V models. It shows similar memory costs among different methods. Table A-5: GPU memory cost of different E2V methods. All tests are conducted on one NVIDIA Tesla 32G-V100 GPU. | Methods | GPU memory | | --- | --- | | E2VID [40] | 12120 | | ET-Net [54] | 12218 | | HyperE2VID [12] | 12227 | | LaSe-E2V (Ours) | 12139 |

Author response to Reviewer H8fU (2/2)

> **Artifacts Not Captured by Traditional Metrics in Figure 3** > - Tab. A-6 provides a quantitative comparison based on three sequences (*horse_11*, *horse_12*, *horse_13*) from the HS-ERGB dataset. Existing E2V methods typically fail to reconstruct regions without events, leading to significantly **worse** quantitative results. Although our method may **not perfectly** reconstruct every detail for reality, it does generate a **reasonable** output that aligns with human preference and is generally **close to the distribution** of the real scene. Therefore, while our results on the HS-ERGB dataset may be less significant than those on "constantly moving" datasets (ECD, MVSEC, HQF), our method is still **substantially better** than baseline methods. - Regarding the artifacts in the letters, as also noted by reviewer ***uWG8***, these issues arise because our method relies on the pre-trained diffusion model (SD2 [42]), which also struggles with text rendering. However, the recently released SD3 [79] claims to have improved text rendering capabilities, which could potentially address this problem and improve the performance. We believe that the MSE can capture these artifacts at **pixel level**. However, although minor artifact in small regions exists, our method excels in **overall** image quality, since previous methods tend to reconstruct misty areas in front of the poster in Fig. 4 (3rd column) Table A-6: Quantitative comparison of HS-ERGB. Results are conducted on 3 sequences with a total 497 frames. | Methods | | HS-ERGB | | | --- | --- | --- | --- | | | MSE | SSIM | LPIPS | | E2VID | 0.199 | 0.382 | 0.736 | | HyperE2VID | 0.161 | 0.374 | 0.745 | | **LaSe-E2V (Ours)** | **0.078** | **0.429** | **0.665** | > **Reference to ESA** > - Our ESA module is specifically designed to enhance spatio-temporal consistency between events and videos. In contrast, the original SD [42] serves as a baseline attention mechanism for integrating conditional input. Our approach differs in **attention design**. While the original SD relies solely on **simple cross-attention** to incorporate various feature conditions, our ESA module takes into account the unique spatial and temporal characteristics of event data. It introduces two **distinct attention** mechanisms respective to the **spatial** domain and **temporal** domain, which fully leverage the constraints provided by the event data and ensure spatio-temporal consistency. **Additional Reference:** [80] Duan H, Min X, Wu S, et al. UniProcessor: A Text-induced Unified Low-level Image Processor[C]//ECCV. 2024. [81] Qi C, Tu Z, Ye K, et al. Tip: Text-driven image processing with semantic and restoration instructions[C]//ECCV. 2024. [82] Gandikota K V, Chandramouli P. Text-guided Explorable Image Super-resolution[C]//CVPR 2024.

Author response to Reviewer ueNs (1/2)

Thanks for bringing up this thoughtful discussion. > **About the hallucinations issue** > - We respectfully disagree with the reviewer. Please note that all reconstructed results are **not unreliable**. In regions with sufficient event data, our method enhances performance with text guidance while ensuring fidelity. As demonstrated in Tab. 1 and Fig. 4, our method significantly **outperforms** previous methods. We also encourage the reviewer to recheck the demo video provided in our supplementary materials. For regions with insufficient event data, our method relies primarily on text prompts to reconstruct a scene that aligns with human preferences. As demonstrated in Fig. 1, HyperE2VID struggles with **severe artifacts** (haze-like artifacts), typically in the background regions, far from the true distribution of real images. Whereas, our method exhibits **higher-quality** reconstruction results that are closer to the ground truth. The results indicate that our method can subtly leverage the semantic information from language and thus ensure semantic consistency of the reconstructed video. To further demonstrate the effectiveness of our method, we provide a quantitative comparison of the HS-ERGB dataset, which includes larger regions lacking event data, as shown in Tab. A-6. Our method substantially **outperforms** baseline methods with an MSE of 0.083. Apparently, the quantitative results verify the superiority and reliability of our method. - In this work, we mainly focus on a new research direction of exploring E2V reconstruction from a language-guided perspective, as affirmed by other reviewers: "This approach is expected to "*inspire future work*" (***ueNs***) and "*provide new ideas*" (***uWG8***). We did not focus much on the computation efficiency of the diffusion model, which will be left as a future work. Please be noted that using language guidance does not actually hamper the application value. Such a text-guided manner is also similar to recent methods used in text-guided denoising [80,81] and super-resolution [82], which have provided broad applications for low-level vision in a user-friendly manner. Our research is **application-significant** as the language provides abundant semantic information, beneficial for ensuring the **semantic consistency** of the reconstructed video. The text prompts also provide a way of human intervention to control the reconstruction process. > **What is the significance of the text in this model?** > - The text information serves as a **crucial** component to **activate** the semantic prior in the diffusion model. As clarified above, our method effectively enhances performance in regions **both** with sufficient and insufficient event data by utilizing text prompts. To simplify text prompts and enhance the model's robustness, we utilize only coarse, tagging-style descriptions for training. Without text prompts, it is challenging to exploit the semantic prior in the diffusion model to ensure semantic consistency. As shown in Tab. 3 (1st row), without the text prompt, our method yields modest results. However, when text information is incorporated, it significantly **improves** the model's ability to leverage the semantic prior in the diffusion model, thereby enhancing performance. > **Text-related parts of the experiments** > - For the qualitative comparison in Fig. 4, we respectfully suggest the reviewer recheck the supplementary **video** material. It shows not only an improvement in contrast. The previous methods tend to reconstruct misty images with distinct artifacts, especially in the region with insufficient event data. Our method effectively exploits the semantic prior of the diffusion model with the guidance of text prompts, which demonstrates semantic consistency. Additionally, Fig. 1 shows another extreme case. While the previous method (i.e., HyperE2VID) fails in the regions without event data, our method still reconstructs a reasonable scene according to the text guidance, which is closer to the ground truth. Tab. A-6 also shows the quantitative comparison in this dataset (i.e., HS-ERGB) and demonstrates the superiority of our method. - Regarding the artifacts in the letter “g”, as also noted by reviewer ***uWG8***, these issues arise because our method relies on the prior from the pre-trained diffusion model (SD2 [42]), which also struggles with text rendering. However, the recently released SD3 [79] claims to have improved text rendering capabilities, which could potentially address this problem and improve the performance.

Author response to Reviewer ueNs (2/2)

> **Regarding the source of the text prompts for the events** > - Our framework primarily focuses on providing a potential pipeline for reconstructing videos from events with a **language-guided perspective**. This is based on our finding that language naturally conveys abundant semantic information, which enhances the semantic consistency of the reconstructed video (see Lines 39-41). Such text guidance manner is **similar** to text-guided denoising [80,81] and super-resolution [82] methods, which also rely on natural language as a user-friendly interface to control the image restoration process. In this work, we do not focus much on the specific sources of the text prompts, as these can be flexible and generic, varying depending on the application scenarios. For example, if a DAVIS346 camera is used, obtaining text from APS frames is convenient. If a Prophesee camera is used and only event data is available, it is feasible to provide tagging-based text prompts derived from event-based multi-modality models. Additionally, in complex scenes lacking sufficient event data, it is practical to incorporate human intervention in an interactive manner to support human control. Overall, our method provides a language-guided interface for E2V reconstruction that could be **feasible** and **generic**. - The reviewer might misunderstand the significance of the text prompts. The text is **not only** used to reconstruct regions without events. For regions with event data, our method incorporates text to provide complementary semantic information, ensuring semantic consistency and further improving performance. For regions without event data, our method can rely on the text to reconstruct a reasonable scene close to the distribution of the real images. The results in Tab. 1 and Tab A-6 have demonstrated the superiority of our method in both scenarios. - Regarding event-based multi-modality models, they offer an option for providing semantic information, which can be explored in future work. Since these models are trained with large-scale event-image paired data, they have the potential to provide prior semantic information that is absent in event data. Based on these text prompts, the method reconstructs video from event data, while effectively exploiting the semantic prior in the diffusion model to ensure semantic consistency. Thanks for the insightful question! We will include this discussion in the camera-ready version.