Enriching Disentanglement: From Logical Definitions to Quantitative Metrics

Summary

This paper investigates relating logical definitions of disentanglement and existing quantitative metrics via formal derivations of novel metrics.

Strengths

The topic is very interesting and it seems the authors were very rigorous in their investigations especially concerning the amount of all of the background material and derivations.

Weaknesses

However, the paper is very poorly motivated, structured and written. It is very difficult to follow the authors along what they are trying to achieve (the motivation), what they are doing (their ideas) and the details on how this is connected with other research (context via related work) and how this should be used for future ML research/ what the significance is of their work for the future. E.g. the Introduction dives right into formal background notations without any motivation/overview of what the authors wish to achieve/investigate in the work. Maybe I have misunderstood some things, but what exactly are the results of the section 3? I.e which of the many metrics are the relevant ones and what do they mean intuitively? The authors provide many examples of specific “implementations” (for lack of a better word here) and then conclude by statements such as “Upon analyzing the metrics above, it becomes evident that what we need is not the best approximation  itself (e.g., the mean) but rather the approximation error “ on page 7. So it seems the previous derivations are irrelevant for the overall goal (what this is is further unclear). So I wonder whether we really require this material then and can remove it, e.g., for the sake of more details on experiments (see below). Overall, I noticed the supplementary materials has much more relevant information that I believe needs to be in the main text. E.g. related works as these are necessary to understand the context of the work. Further, it was also very difficult for me to understand the experiments of the main paper. The experimental setup is not described in the main paper (there is not even a caption to Table 2). This needs to be greatly improved. There seem to be a lot of information in the supplementary concerning experiments and I would suggest moving this into the main paper. Overall, it would help me if the authors could summarize again what they were investigating with these experiments. The claims made in the experiments of the main paper are not backed up by the experiment shown in Table 2, e.g. "The metrics derived from equivalent definitions may differ in terms of computation cost and differentiability.". Where are the results for this? If these can only be found in the supplementary currently, I think they should also be moved to the main text. I am also puzzled by all of the experiments in the supplementary material (on interesting datasets), but that are not mentioned in the main text (unless I have missed this). Was there a reason for this?
 Lastly, I am missing a thorough discussion of the overall findings and particularly potential limitations of the findings/proposed metrics. Overall, it is too difficult to assess the quality of the work and proposed ideas with the current structure. I would recommend putting some time into rewriting and restructuring to make this more clear to the reader.

Questions

see above

Rating

5

Confidence

3

Soundness

2

Presentation

1

Contribution

3

Limitations

see above

Summary

The paper consider the connection between logical definitions and quantitative metrics, proposing a systematic approach to design metrics from logical definitions. Particularly, the paper is focused on the measure of disentanglement. The paper theoretically justifies the correspondence between logical definitions of disentanglement and quantitative metrics via topos theory and enriched category theory and empirically demonstrates the effectiveness in isolating various aspects of disentangled representations compared to existing metrics.

Strengths

The idea presented in the paper, especially the systematic method of converting logical predicates into quantitative metrics, is novel and very interesting. The authors also theoretically support this idea with topos theory and enriched category theory, which are sophisticated and advanced mathematical frameworks not commonly employed in machine learning literature.

Weaknesses

The superiority of the proposed metrics over the the existing ones does not seem fully validated, either empircally or theoretically. The evidence provided does not show the proposed metrics dominate the existing ones universally.

Questions

Could you provide more evidence, either empirical or theoretical, to show the proposed metrics dominate the existing ones universally? Or if the proposed metrics do not outperform the existing ones universally, when are they better?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper proposed to establish a connection between logical definitions of disentanglement and quantitative metrics from the perspective of typos theory and category theory. It then propose a metrics for disentanglement with stronger theoretical guarantees and compared it with some state of the art metrics.

Strengths

The paper provided theoretical justification for establishing metrics from typos theory and category theory in disentanglement of representation learning. It then proposed to convert first-order predicate into real-value quantity, which is innovative. The proposed metrics also have practical implications as the author mentioned differentiability and its effectiveness through some experimental results.

Weaknesses

1, Despite that the authors included as much background information in the appendix, the advanced mathematical concepts makes the paper very hard to follow and limit its accessibility. Moreover, the theoretical connections may not be readily feasible to practical and intuitive interpretations. Hence its use scenarios should be stated. 2, Even though the paper provides empirical results, its scope and effectiveness may not be fully demonstrated through the limited range of scenarios considered. Therefore its generalizability it not clear.

Questions

1, why were the specific modularity metrics used for evaluation? How do they contribute to assessing disentanglement in representation learning? 2, Line 308 mentioned that the results are transformed isomorphically using $e^{-x}$. What is the purpose of this procedure?

Rating

5

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors adequately addressed the limitations

Summary

This study introduces a systematic approach to quantify properties of representation learning models. By translating logical definitions into quantitative metrics, the paper evaluates two key properties: modularity and informativeness. Two sets of metrics are derived for each property, one based on approximation and the other on distance computation and aggregation. Theoretical analysis of these metrics is conducted, including examination of their minimizers. (Full disclosure: I have reviewed this paper before)

Strengths

1. I think this paper is interesting and innovative and mathematically sound. 2. The paper is well-written. The math objects are introduced with good intuitions.

Weaknesses

1. It might be difficult for people with less prior knowledges to read. 2. It’s unclear if the metrics is practical be to a loss to optimize in practice because the experiments do not train with proposed metrics.

Questions

1. Heyting algebra is mentioned a couple of times in the paper without a proper definition, which would be good for general ML audience.

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A