Language Models as Hierarchy Encoders

Summary

This paper proposes a method that leverages a non-Euclidean representation space to better encode hierarchical relationships between entities. Specifically, a hyperbolic embedding space is defined, wherein items closer to the origin are higher-level concepts, and items further from the origin are lower-level concepts; items closer to each other typically have closer semantic relationships, such that it is then straightforward to translate between geometric distances and inheritance relationships between concepts—even when the relationship was not explicitly encoded in the training set. In a series of experiments, the authors leverage sentence-Transformer models as bases to predict whether two entities have taxonomic relationships in a zero-shot manner. Comparisons are performed between the base sentence-Transformer models, the same models with task-specific fine-tuning, and the proposed hyperbolic encoding method. It is observed that the proposed method outperforms more naive methods.

Strengths

* The proposed embedding space works well as a way to categorize hierarchical relationships between inputs. It also makes it easy to tell what the relationship is between two concepts in latent space, as distances between concepts and distances to the origin both have well-defined meanings in this space. * The method can be run with modest compute resources, making it accessible and easily scalable as sentence-Transformer models scale and improve. * The idea is interesting and, to my knowledge, novel.

Weaknesses

1. As the authors acknowledge, it is not clear what other knowledge is lost when moving to this new latent space. While hierarchical relationships between entities are now easy to understand, does the new space also preserve properties of each independent entity? Evaluations on a wider variety of downstream tasks would be helpful for establishing what kinds of semantic information are preserved. 2. The analysis in Table 4 could have been cherry-picked. A more systematic version of the analysis in Table 4 is possible, and would be nice to see: specifically, one could search over a larger subset of WordNet, quantifying whether more deeply nested concepts correlate with higher h-norms, and whether more closely related concepts have significantly lower hyperbolic distances than unrelated but semantically similar concepts. 3. When we classify whether two entities are taxonomically related, we usually care more about whether we can leverage these relationships to improve performance on some other metric. For example, better representational quality could lead to better deductive reasoning. Consider, for example, a counterfactual reasoning task where we say “Birds have fur. Is it therefore true that sparrows have fur?” A model that better captures taxonomic relationships should be better at this task. What I’m getting at is that it would be nice to evaluate on downstream tasks that require entity classification as a key part, rather than directly evaluating entity classification (a rare task). This would demonstrate the utility of this method, and increase interest and impact among a broader audience.

Questions

1. Why sample to a consistent 1:10 ratio of positive:negative examples? Was this decision made empirically? 2. How long did it take to run your proposed method compared to standard fine-tuning? Is there a better trade-off between performance and compute?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

The paper introduces a method to re-train transformer-based language models as Hierarchy Transformer encoders (HITs), using the properties of hyperbolic space to enhance their ability to encode hierarchical structures in language.

Strengths

1. The utilization of hyperbolic space to encode hierarchical structures in language models is a creative and theoretically sound approach, as hyperbolic space naturally lends itself to representing hierarchies. 2. The paper provides extensive experimental results, showing that HITs consistently outperform traditional pre-trained and fine-tuned models across multiple datasets and tasks. 3. The methodology, experimental setup, and results are clear, making the paper accessible to readers.

Weaknesses

1. The motivation is a bit overstated since it is heavily on the claim that current language models are significantly deficient in encoding hierarchical information. Thus, improvements can certainly be made. 2. The application of hyperbolic spaces in language models, while interesting, isn't entirely quite inspiring as previous works have explored similar ideas. The distinction between this and prior approaches isn't as significant as suggested. 3. The downstream tasks selected by this study have deviated from those for the LLMs.

Questions

See weaknesses.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper proposes a new way to retrain encoder-based language models into hierarchy encoders. Specifically, they propose to recast the output embedding space onto a Poincaré ball and retraining with the designed loss functions for organizing entities into hierarchy. The experiments on real world datasets like WordNet demonstrates the advantages of the proposed approach.

Strengths

- The paper is well-written, scoped, and nicely organized. - The topic of addressing hierarchy with language models is interesting. - The proposed approach is novel and the corresponding experimental results validate the effectiveness of the proposed method.

Weaknesses

- Can you explain if the proposed method can be used to decoder-only models? - Will frequency of entities in the data impact the hierarchy captured in the representations?

Questions

- see. weakness

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discuss their limitation in the sec 6.

Summary

This paper presents a novel approach called Hierarchy Transformer encoders (HITs) to retrain transformer-based language models to better encode hierarchical structures in language. The method involves situating the output embedding space of pre-trained language models within a Poincaré ball and training on hyperbolic clustering and centripetal losses. The authors evaluate HITs against pre-trained LMs, fine-tuned LMs, and hyperbolic embedding baselines on tasks including multi-hop inference and mixed-hop prediction. Results show that HITs consistently outperform baselines, demonstrating improved ability to capture hierarchical relationships and generalize across hierarchies.

Strengths

1. The paper introduces a novel method to explicitly encode hierarchies in language models without adding new parameters. 2. The authors conduct extensive experiments across multiple tasks and datasets, and show the effectiveness of their approach.

Weaknesses

1. The proposed method requires predefined hierarchies which may limit its generalization, especially when new entity arrives while no relation is presented for the new entity. 2. Applications of the learnt entity embedding are desired to show the effectiveness of the embedding. For example, how can the embedding be used in downstream tasks.

Questions

Could you show more examples on how hierarchies are encoded through the proposed approach?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

thank you for the rebuttal.

thank you for your response.