Thank you for your recognition of our work and your thoughtful considerations. Let me answer the question:
1. **Difference between Cosine, Dot and Euclidean Distance Similarity in Gradient-Based Methods**: As you pointed out, the difference cosine, dot product, and Euclidean distance similarity can be subtle, and in practice, the choice of similarity metric often depends on experimental results (In our experiments, we found that CHG Shapley outperforms both Gradient-dot and Gradient-cosine in identifying high-value data and distinguishing noisy data scenarios.).
2. **Shapley Value under Different Similarity Metrics**: Although the differences between the similarity measures are small, incorporating these into the **Shapley value computation** results in very different forms for each similarity metric:
- The dot product-based Shapley value still similar to the dot product form, though it requires subtracting a constant (see Appendix B, derivation of $ \phi_j(U_2) $).
- The Euclidean distance corresponds to the formulation in Equation (4).
- However, for cosine similarity, it is really hard to compute the Shapley value due to norm-related issues and I'm almost sure that it's impossible to derive a clean solution for this case. This is an open problem.
3. **Upper Bound of Training Loss vs. Approximation**: The theoretical foundation of the paper aims to estimate the change in training loss when applying gradient descent to the current network parameters across different subsets of the training data. The **upper bound** of $ f(\theta_x) - f(\theta) $ serves as a **conservative approximation** of the training loss reduction (**conservative** means the upper bound holds for any $\theta_x$). The reason I didn't use an approximation is that I initially considered commonly used optimization bounds but hadn't thought of applying a Taylor expansion in this context :)
4. **Gradient Similarity as a Data Valuation Method**: I totally agree that gradient similarity can serve as a heuristic for data valuation (and the mentioned papers indeed provide some evidence), but as pointed out on 2, I am cautious about framing this method as a strict data valuation technique because they do not directly compute Shapley values, as in the "Data Shapley: Equitable Valuation of Data for Machine Learning" paper. Therefore, to maintain theoretical rigor, I prefer to avoid calling it a "data valuation" method in the strictest sense though they have been used to measure the data value in practice.
5. **Why is the sample size S so important?**
- **From Machine Learning View**: The division $S$ actually represents the operation of taking the expectation. In traditional machine learning theory, the average loss over $S$ samples (i.e., empirical risk) approximates the generalization risk. This approximation is foundational for theoretical bounds in machine learning, such as Hoeffding’s inequality, which quantifies the relationship between empirical risk and generalization error. Practically, almost all machine learning algorithms compute the average loss over all available samples.
- **From Cooperative Game Theory view**: The devision of sample size $S$ affects the accuracy of Shapley value computations. Consider 5 samples (4 good, 1 noisy), with gradients $ g_i $ and the model update direction $ g_{\text{mean}} $. The gradient inner products are [0.6, 0.6, 0.6, 0.6, 0.1], where the noisy sample’s gradient is less aligned with the correct direction. Using a first-order Taylor expansion for the utility function $ U(S) = \sum_i \langle g_i, g_{\text{mean}} \rangle $, the Shapley values are [0.6, 0.6, 0.6, 0.6, 0.1]. In contrast, using the rigorer form utility function as in this paper ($ U(S) = \frac{1}{S} \sum_i \langle g_i, g_{\text{mean}} \rangle $) results in Shapley values of [0.152, 0.152, 0.152, 0.152, -0.108]. Typically, negative Shapley values indicate low-quality samples (In the experiment shown in Figure 3 of [1], negative Shapley-value samples were also filtered out to get the "clean" datasets) , but the first method assigns 0.1 to the noisy sample, while the second assigns -0.1. This demonstrates that neglecting the sample size $S$ leads to inaccurate Shapley value estimation.
6. **Naming "GradE Shapley"**: Thank you for pointing out the potential confusion with the term "Gradient Shapley," as it has been used in prior work. While Algorithm 2 in Ghorbani and Zou’s paper and our approach are conceptually quite different (the former affects model training process because of the Monte Carlo sampling of data subsets, while our method does not (see algorithm 1)), we recognize the need for clearer terminology. Given this overlap, I have revised the name to "GradE Shapley" (Gradient Euclidean Shapley) to better reflect the specific approach and avoid any confusion with the earlier work.
Thank you once again for your valuable insights. Your suggestions help improve the clarity and rigor of our work.