Pearson's r, the most widely-used correlation coefficient, is traditionally regarded as exclusively capturing linear dependence, leading to its discouragement in contexts involving nonlinear relationships. However, recent research challenges this notion, suggesting that Pearson's r should not be ruled out a priori for measuring nonlinear monotone relationships. Pearson's r is essentially a scaled covariance, rooted in the renowned Cauchy-Schwarz Inequality. Our findings reveal that different scaling bounds yield coefficients with different capture ranges, and interestingly, tighter bounds actually expand these ranges. We derive a tighter inequality than Cauchy-Schwarz Inequality, leverage it to refine Pearson's r, and propose a new correlation coefficient, i.e., rearrangement correlation. This coefficient is able to capture arbitrary monotone relationships, both linear and nonlinear ones. It reverts to Pearson's r in linear scenarios. Simulation experiments and real-life investigations show that the rearrangement correlation is more accurate in measuring nonlinear monotone dependence than the three classical correlation coefficients, and other recently proposed dependence measures.
Paper
Similar papers
Peer review
Summary
The authors here introduce a new correlation statistic here they call the "rearrangement correlation:" $r^{\sharp}$. The work argues this statistic captures non-linear, monotonic relationships between two samples. The authors prove a couple theoretical results for this new statistic and then conduct a large empirical analysis comparing this statistic to many alternatives that measure the strength of relationship between two variables.
Strengths
The paper provides a simple summary statistic (called the rearrangement correlation $r^{\sharp}$) that aims to capture correlation between two variables that may just be monotonic, rather than linear. This statistic is simple to compute (requires $O(n \log n)$ to sort each sample) and is 1 if and only if one sample is increasing monotone dependent in the other. They also show that it is bounded below by the Pearson correlation statistic. The authors provide some empirical analysis to suggest that this statistic is well-suited to capture the coefficient of determination between two samples even when the relationship is non-linear. They perform this analysis with both simulated and real-world data and compare their new statistic to many other alternatives. Originality: [+] + The idea of introducing a different correlation statistic is not completely new, but to the best of the reviewers knowledge, this statistic is not known. This appears to be a new statistic. + The authors are able to lower bound this new statistic with the Pearson correlation statistic, which is a more well known statistic. Quality: [+] + The paper is mostly well-written and easy to follow. + Figure 1 does a nice job illustrates a hierarchy of statistics in the case that the correlation coefficient is 1. Clarity: [+] + Most of the paper is easy to follow, barring some parts of the experimental section. Significance: [+] + The authors have introduced a new correlation statistic that appears to encode some mutual information between two paired samples. + When the relationship between two samples is roughly monotonic, the statistic appears better then many alternatives to estimates the $R^2$ between those two samples.
Weaknesses
The main weakness in the current form of the paper is when to utilize this statistic. The paper mentions that it is best as a proxy for "strength measurement," but its main utility seems to be using it to measure the strength of a monotonic relationship between two artibrary samples. This is useful, but if this is the core utility, it is not clear how much better it is compared to using the RSS of a LOWESS regression or some summary statistic of a QQ-plot. If there were more theory on the topology of this statistic, one may be able to use it as a measure of how well a model fits a dataset, i.e., instead of the Pearson coefficient of $y$ and $\hat{y}$ (which is basically the $R^2$ for the model) one could compute $r^{\sharp}$ instead. However, there is no real analysis in the paper of whether a larger $r^{\sharp}$ actually means it is a better fit in some sense. Overall, I think there is something interesting about this statistic, but it feels a bit unfinished. E.g., understanding robustness to outliers, how sensitive it is if one sample changed monotonically, or if it could be used in hypothesis testing would be huge reasons to consider this statistic more broadly. Originality: [-] - Outside the introduction of this statistic, much is still relatively unknown. E.g., what is it limiting distribution as $n$ goes to infinity? Is it robust to outliers? Is it sensitive to monotonic transformations? Quality: [-] - It would be nice to move the proof of Theorem 1 to the appendix, as it isn't really necessary to understand the rest of the paper and there already are proofs in the appendix. - What exactly does the population value $|r^{\sharp}(X, Y)|$ measure? Is there any intuitive way to understand its value? Clarity: [-] - The experiment in Section 3 is not described in full detail. Is the experiment computing correlations between $f(x)$ and $y$? Is it computing correlations between $x$ and $y$? It would be helpful to spell this out a bit more clearly. I think its the latter but it would be better to be explicit. - Section 3.1 spends a lot of time discussing "trueness" and "precision," when it suffices to just suggest MAE as a statistic. There is no real need to explain bias and variance to this audience :) Significance: [-] - The authors mention is it meant to be used as a "strength measurement," but the context for when to use this measurement is not entirely clear. E.g., the authors demonstrate that it can be used to measure the amount of noise present between two samples $x$ and $y$ where $y$ is roughly a (monotone) function of $x$, but if this is the use case, why couldn't one just use a locally-weighted regression (e.g. LOWESS) to estimate $f$ and thus the $R^2$? The true benefits of using this statistic are still a bit unclear. - There are still some questions that remain regarding this statistic: e.g., is there some proof that explains roughly under what $f$, $\sigma$, error distribution, etc. will yield reasonable estimates for the $R^2$? Detailed Comments: L36: No need to mention the title of the paper, you can just mention the authors and provide the citation. L51-56: IMHO, most of these definitions are common enough to omit. One could just state "X and Y are assumed to be random variables with bounded second moment."
Questions
Q1: Given this measure is better suited to handle non-linear relationships between data, how does it compare to computing the Pearson correlation between the quantiles of two samples? Or, e.g, does computing some summary statistic of a QQ-plot encode some of the same information being picked up by $r^{\sharp}$? Q2: How does the new correlation measure $r^{\sharp}$ encode different relationships in data? E.g., if we have three datasets $x = (x_1, ..., x_n)$, $y^1 = (y^1_1, ..., y^1_n)$ and $y^2 = (y^2_1, ..., y^2_n)$, what is implied if $r^{\sharp}(x, y^1) > r^{\sharp}(x, y^2)$? Does this signify a stronger relationship between $x$ and $y^1$ than $x$ and $y^2$? How should one really interpret this for correlations not close to 1? Q3: In what situations should one prefer this statistic over another? E.g., for approximately linear relationships, the Pearson correlation should be better than this measure. In what settings and under what assumptions should the reader opt to use this statistic?
Rating
3
Confidence
3
Soundness
3
Presentation
3
Contribution
2
Limitations
NA
I thank the authors for their response. In particular, thank you for illustrating the connection between $r^{\sharp}$ and the quantiles of the samples, when to use this statistic over the Pearson correlation (but not Kendall-tau), and the permutahedron is an interesting idea. Unfortunately, I still do not feel any of my main questions have been addressed: * In the section responding to LOWESS vs. the new method, the authors simply state that "correlation and regression are different." I'm not totally sure what the purpose of the distinction is, I'm merely asking why is this any better or different than using LOWESS to derive a statistic to encode the same information as $r^{\sharp}$. The authors didn't really address this question. * The authors did not provide any insight about the topology of $r^{\sharp}$. E.g., when asked if a larger $r^{\sharp}$ indicates a better fit, the authors just pointed toward the empirical work and claimed that it was the case without any proof. My question is a theoretical one in nature and can not be proven by evaluating a few examples from the synthetic datasets being used in the empirical section. * I don't think the authors answered my question about what in the population setting $r^{\sharp}(X,Y)$ is measuring. * For some of the core principles needed in statistical theory (e.g., limiting distribution, robustness to outliers) and also ones specific to this statistic (sensitivity to monotonic transformations), no further analysis was provided. If one is to propose a new statistic, some analysis of these is crucial, otherwise it is difficult for practitioners to put any trust into this statistic over another. E.g., the $R^2$ of the LOWESS regression above is at least very well understood in terms of statistical theory, so to use $r^{\sharp}$ over that statistic, there must be some theoretical discussion of why this is the case. For these reasons, I am inclined to keep my score on this paper.
1. Correlation analysis and regression analysis fulfill distinct roles, as highlighted in our previous responses, and thus, they are not directly comparable. Indeed, none of those renowned dependence measures, such as distance correlation (Székely, Rizzo, & Bakirov, 2007), Maximal Information Coefficient (MIC) (Reshef et al., 2011), and Chatterjee's ξ (Chatterjee, 2021), are intended for comparison with other regression methods. 2. LOWESS regression has been mentioned several times. However, to the best of our current understanding, it does not exhibit any particular association or relevance to monotonic dependence. 3. In our manuscript, we present the underlying motivation, delve into the core theoretical issues, and report on the experimental performance. The story is complete. 4. As a newly proposed method, it poses a challenge for us to fully explore and elucidate all of its theoretical properties within the confines of a single paper. 5. The problem you mentioned has inspired us greatly. We will explore it further in future works. Concise answers to the problem are as follows: - Is it robust to outliers? NO. Rearrangement correlation is a scaled covariance, and the limitation of being non-robust to outliers is inherited from covariance itself. In fact, concordance correlation coefficient, additivity coefficient, and Pearson's r are also scaled covariance measures, and none of them are robust to outliers. Whether a dependence measure is robust to outliers or not does not solely determine its value or usefulness. --- Székely, Gábor J., Maria L. Rizzo, and Nail K. Bakirov (2007). Measuring and Testing Dependence by Correlation of Distances. In: Annals of Statistics 35.6, pp. 2769–2794. Reshef, David N. et al. (2011). Detecting Novel Associations in Large Data Sets. In: Science 334.6062, pp. 1518–1524. Chatterjee, Sourav (2021). A New Coefficient of Correlation. In: Journal of the American Statistical Association 116.536, pp. 2009–2022.
comparison to spearman rho and kendal tau
Dear authors, thank you for the interesting paper. Given that spearman rho and kendall tau are extremely popular measures of nonlinear ( and monotone) dependence, widely used in life sciences, and also involving ranking operations in their construction, please explain how do they compare with the proposed rearrangement coefficient. In what situations is rearrangement coeff significantly better than spearman rho? Please provide some intuition why. Without contrasting wrt these standard measures the paper falls short of its potential impact. Thank you.
I am deeply appreciative of your insightful comments. ${r^♯}$ is better than the classical Spearman's $\rho$ in sense that: 1. ${r^♯}$ has a higher *resolution* and is more accurate. Notably, all measures devised specifically for monotonic dependence inherently rely on order information. Yet, our approach uniquely leverages the original data points, foregoing the conventional ranking process. The conversion of numerical values to ranks inherently entails a loss of information, as subtle differences between values may become indistinguishable from more pronounced disparities. Given a sample size of $n$, Spearman's $\rho$ can only assume a finite set of $\frac{{{n^3} - n}}{6}$ distinct values within the range of -1 to +1, regardless of the underlying raw values or the specific correlation patterns. Consequently, the granularity or *resolution* of Spearman's $\rho$ may be considered suboptimal, potentially obscuring nuanced differences in the data. To illustrate this concept with a concise yet illustrative example, let - $x = \left( {4,3,2,1} \right)$ - ${y_1} = \left( {5,4,3,2.00} \right)$ - ${y_2} = \left( {5,4,3,3.25} \right)$ - ${y_3} = \left( {5,4,3,3.50} \right)$ - ${y_4} = \left( {5,4,3,3.75} \right)$ - ${y_5} = \left( {5,4,3,4.50} \right)$ It is evident that ${y_1}$ and $x$ exhibit identical monotonic behavior, with their values decreasing in a stepwise fashion. Conversely, the behavior of ${y_2}$, ${y_3}$, ${y_4}$, and ${y_5}$ diverges increasingly from that of $x$ as the final element deviates further from the strict descending pattern. However, when assessing the relationships using Spearman's $\rho$, the values for ${y_2}$, ${y_3}$, ${y_4}$ are identical. This underscores a limitation of Spearman's $\rho$ in distinguishing subtle variations in monotonic relationships, particularly when the differences are confined to the ranks. In stark contrast, ${r^♯}$ is capable of precisely capturing these differences. It reveals the gradual shift in the relationship between $x$ and $y$, thereby providing a more accurate and informative assessment of their monotonic dependence. - ${r^♯ }\left( {x,{y_1}} \right) = 1.00$, $\rho \left( {x,{y_1}} \right) = 1.00$ - ${r^♯ }\left( {x,{y_2}} \right) = 0.93$, $\rho \left( {x,{y_2}} \right) = 0.80$ - ${r^♯ }\left( {x,{y_3}} \right) = 0.85$, $\rho \left( {x,{y_3}} \right) = 0.80$ - ${r^♯ }\left( {x,{y_4}} \right) = 0.76$, $\rho \left( {x,{y_4}} \right) = 0.80$ - ${r^♯ }\left( {x,{y_5}} \right) = 0.38$, $\rho \left( {x,{y_5}} \right) = 0.40$ Experiments conducted in both simulated and real-world scenarios have also conclusively demonstrated that ${r^♯}$ exhibits a higher degree of accuracy compared to $\rho$, underscoring its superiority in capturing intricate monotonic relationships. 2. ${r^♯}$ is comparable with Pearson's $r$, while the latter is not. For nonlinear monotonic dependence, the value of Spearman's $\rho$ might be remarkably greater than the value of Pearson's $r$. One may attempt to search for nonlinear relationships in data by checking whether the value of $\rho$ far exceeds that of $r$. However, it might be meaningless and even impossible to compare their values directly. It is possible for $\rho$ to be either greater or smaller than $r$, and their signs may differ, rendering the difference $\left| \rho \right| - \left| r \right|$ ambiguous. In contrast, the signs of ${r^♯}$ and $r$ are consistently aligned, and the magnitude of $\left| {r^♯} \right|$ is always greater than or equal to $\left| r \right|$. Specifically, $\left| {r^♯} \right| - \left| r \right|$ equals to 0 if and only if $y$ is arbitrary permutation of $ax + b$. Furthermore, its value increases with the degree of nonlinearity. --- Spearman's $\rho$ can also be superior to ${r^♯}$ in sense that the former is robust to outliers, while the latter is not. In order to be more robust, we can also transform the raw data into their ranks before calculating ${r^♯}$. And we have made an intriguing discovery. 3. ${r^♯}$ reverts to Spearman's $\rho$ if calculated on ranks. We have derived that: if $x$ and $y$ are both permutations of {1, 2, ..., $n$}, then we have ${r^♯}\left( {x,y} \right) = \rho \left( {x,y} \right)$ in sense that $s\left( {x,y} \right) = s\left( {{x^ \uparrow },{y^ \updownarrow }} \right) = \frac{{n\left( {n + 1} \right)}}{{12}}$. The aforementioned conclusion sheds light on the following phenomenon: Spearman's $\rho$ is simply the application of Pearson's $r$ in the data converted to ranks before calculating the statistic. Now that $\rho$ and $r$ share the same mathematical formula, why can the former measure nonlinear monotonic relationships while the latter only linear ones? The crux is not the conversion of raw data to ranks, but still to have a sharp bound. For ranks, $\rho $ and ${r^♯}$ are equivalent. Since we have established in our manuscript that ${r^♯}$ can measure arbitrary monotone dependence, $\rho $ shares this capability.
please reflect this in the paper
Thank you for the clear explanation — this is helpful. I would strongly recommend to add an abridged version of this discussion to the paper, it adds context and intuition. Perhaps instead of the accuracy and precision discussion, which may be redundant for NeurIPS audience.
Thank you very much for your advice. We will revise the paper accordingly and incorporate an abridged version of this discussion into the manuscript. The invaluable insights you have shared, coupled with the contributions from four esteemed reviewers, have been instrumental in significantly enhancing and refining our work. We are profoundly grateful for your guidance and support throughout this process.
Thank you for the response. Per your comments: I am not claiming that regression and correlation aren't different. I am suggesting that looking at the percentage of variance explained by a LOWESS regression of two variables is a plausible approach to measure the strength of correlation, but in the current paper is not considered as an alternative method. This seems like a very reasonable way to capture the relationship between variables when monotonicity is the focus and it would prove the utility of this correlation to compare to this as benchmark. When I discuss the topology of $r^{\sharp}$, the only thing that seems to be proven in the text is that $r^{\sharp}(x, y) = 1$ iff the discrete samples $x = (x_1, \dots, x_n)$ and $y=(y_1, \dots, y_n)$ are monotone increase w.r.t to each other. But consider an example where $0 < r^{\sharp}(x, y) < 1$. If $f$ is some strictly monotone increasing function, is it even clear how $r^{\sharp}(x, f(y))$ relates to $r^{\sharp}(x, y)$? Given your previous claim that $r^{\sharp}(x, y) = r(x, y) / r(q_x, q_y)$, you should be able to relate this to the pearson correlation if you can prove the above equality is true. Understanding what $r^{\sharp}(x, y) > r^{\sharp}(x, z)$ implies about $y$ and $z$ is another example. These are the types of properties that would provide some better understanding of the metric for practitioners to know why they should use this new proposal. I understand that the paper can only be so long, but there are lots of sections of the paper that would be much better served answering these questions than the current ones posed in the text. As I mentioned in my initial review, I think there could be some interesting ideas in this work, but it seems like there are enough missing details that this paper would appeal to a larger audience if some of the above questions were answered.
Thank you very much for your constructive suggestions and meticulous comments, which have been instrumental in enhancing the quality of our work. We shall endeavor to incorporate within our paper a comprehensive account of the potential theoretical properties pertaining to the proposed $r^\sharp$. For example, 1. We can express $r^\sharp$ in terms of Pearson's $r$ as follows: ${r^\sharp }\left( {x,y} \right) = \frac{{{s_{x,y}}}}{{\left| {{s_{{x^ \uparrow },{y^ \updownarrow }}}} \right|}} = \frac{{\frac{{{s_{x,y}}}}{{\sqrt {{s_{x,x}}{s_{y,y}}} }}}}{{\left| {\frac{{{s_{{x^ \uparrow },{y^ \updownarrow }}}}}{{\sqrt {{s_{x,x}}{s_{y,y}}} }}} \right|}} = \frac{{r\left( {x,y} \right)}}{{\left| {r\left( {{x^ \uparrow },{y^ \updownarrow }} \right)} \right|}}$ 2. $r^\sharp$ possesses a notable invariance property under transformations of location and scale applied to the two variables. Specifically, it remains unaffected by linear transformations of the form $x \mapsto a + bx$ and $y \mapsto c + dy$, where $a$, $b$, $c$, and $d$ are constants with the constraints $b,d > 0$. This invariance highlights the robustness of $r^\sharp$ to common preprocessing steps involving shifts in location and rescaling of the variables. Nevertheless, it is imperative to acknowledge that, in contrast to rank-based correlation coefficients, $r^\sharp$ does not demonstrate invariance under monotonic transformations.
Summary
The authors refined Pearson’s r, and proposed a new correlation coefficient, i.e., rearrangement correlation. They showed that this coefficient is able to capture arbitrary monotone relationships, both linear and nonlinear ones. With simulation, they showed the rearrangement correlation is more accurate in measuring nonlinear monotone dependence than the three classical correlation coefficients, and other recently proposed dependence measures.
Strengths
The introduction of this correlation coefficient, i.e., rearrangement correlation, can measure some nonlinear dependence unlike the classical Pearson's. The simulation also showed it performs better than some other correlation coefficient in the literature.
Weaknesses
The paper's weakness lies in some aspects. First, the contribution of this rearrangement correlation doesn't seem significant enough or at least the authors can clarify more on this. For example, Theorem 1, which was already proposed in the existing literature, seems enough to motivate this rearrangement correlation thus lacking some significant theoretical contribution. In other way, in terms of application, the paper failed to clarify the importance of this rearrangement correlation in some meaningful statistical problems/machine learning algorithms other than just introduce this new concept.
Questions
As mentioned above, Theorem 1, which was already proposed in the existing literature, seems enough to motivate this rearrangement correlation. I am not clear about the authors' significant contributions.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
yes
Summary
The paper proposes an adjustment to Pearson’s r to measure nonlinear monotone relationships, resulting in a new coefficient called the rearrangement correlation. The rearrangement correlation can capture both linear and nonlinear monotone relationships more accurately than traditional measures.
Strengths
This paper rresents a new inequality tighter than the Cauchy-Schwarz Inequality. The proposed Rearrangement Correlation is well-defined but could use more intuitive explanations.
Weaknesses
1. It would be better to analyze some special cases under which how much the proposed correlation can improve the traditional Pearson one. 2. The title is confusing. Linear monotone dependence also fits in the paper's scope.
Questions
1. It would be better to analyze some special cases under which how much the proposed correlation can improve the traditional Pearson one. 2. The title is confusing. Linear monotone dependence also fits in the paper's scope.
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
1. It would be better to analyze some special cases under which how much the proposed correlation can improve the traditional Pearson one. 2. The title is confusing. Linear monotone dependence also fits in the paper's scope.
Summary
This paper proposes a new variant of Pearson's r coefficient to capture arbitrary monotone relationships between random variables. Traditionally, Pearson's r is used to capture linear dependence between variables. The relation of "being linearly dependent" is stronger than the relation of "being monotone dependent (i.e. (X, Y) almost surely lies in a non-decreasing or non-increasing subset of R^2). The capturing power of linear relations by Pearson's r coefficient is explained by equality condition in Cauchy-Schwarz inequality. The authors of this paper demonstrate that strengthening of Cauchy-Schwarz inequality using rearrangement theorems by Hardy, Littlewood and Polya provides a new adjusted Pearson's coefficient, which authors call rearragement correlation coefficient that is able to capture monotonic dependences.
Strengths
The problem studied in this paper is quite natural and has various practical applications. Prior work proposed modifications to Pearson's r coefficient but "in opposite direction" to capture more strict relations: concordance correlation coefficient (Lin 1989) to measure identical relationship, additivity coefficient (Zegers 1986) to capture additive relations Y = +-X+b. This work instead moves in the opposite direction and proposes a coefficient to capture more "loose" monotone relation. Therefore, this paper can be seen as a continuation of that line of work. The paper is well-writtena and clearly presented. Proofs are simple, elegant and easy to follow. Theoretical results are supported by experiment on simulated data and real-world datasets.
Weaknesses
-
Questions
Can this method be generalized beyond one-dimensional data? I.e., is there an analog of "rearrangement correlation coefficient" for a proper definition of monotone dependency between X and Y in R^d?
Rating
7
Confidence
3
Soundness
3
Presentation
4
Contribution
3
Limitations
-
Response
Thank you for the clarification. What I meant is to find special cases and prove theoretical improvement. Not numerical.
Thanks for the clarification on the main contribution. I now understand more through the reply "our novel contribution stems from uncovering the fact that the Rearrangement Inequality offers a tighter bound for covariance compared to the Cauchy-Schwarz Inequality. This tighter bound, in turn, paves the way for the introduction of the novel correlation". Theoretically, it becomes more clear to me so I decided to raise the score a bit. However, I still find it hard to find some special cases on how important it "paves the way for the introduction of the novel correlation".
Decision
Accept (poster)