Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space

Modern generative models demonstrate impressive capabilities, likely stemming from an ability to identify and manipulate abstract concepts underlying their training data. However, fundamental questions remain: what determines the concepts a model learns, the order in which it learns them, and its ability to manipulate those concepts? To address these questions, we propose analyzing a model's learning dynamics via a framework we call the concept space, where each axis represents an independent concept underlying the data generating process. By characterizing learning dynamics in this space, we identify how the speed at which a concept is learned, and hence the order of concept learning, is controlled by properties of the data we term concept signal. Further, we observe moments of sudden turns in the direction of a model's learning dynamics in concept space. Surprisingly, these points precisely correspond to the emergence of hidden capabilities, i.e., where latent interventions show the model possesses the capability to manipulate a concept, but these capabilities cannot yet be elicited via naive input prompting. While our results focus on synthetically defined toy datasets, we hypothesize a general claim on emergence of hidden capabilities may hold: generative models possess latent capabilities that emerge suddenly and consistently during training, though a model might not exhibit these capabilities under naive input prompting.

Paper

References (100)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer o8es8/10 · confidence 4/52024-06-18

Summary

The paper introduces a novel framework for understanding concept learning in text-conditioned generative models. The experiments are carefully designed (although most of them look a bit toyish) and support the analyses well. The three key findings, i.e., concept signal levels determine the learning dynamics; capability consistently emerges before observable behaviors; under-specification harms the system, are quite novel and could provide many insights to the community. The proposed theoretical formulation also has the potential to be further developed. I enjoy reading this paper and suggest an acceptance.

Strengths

- Good presentation, easy to follow. - The discussions about the learning dynamics of concept memorization and compositional generalization are cool. - The three probing methods that verify the claim that the model might learn the concept before generating the correct image are insightful. - The under-specification discussion might bring practical suggestions to real systems.

Weaknesses

- In Section 3, it is a little hard to understand the relationships between different variables and functions. A diagram like Figure 1 in [2] might be helpful. - Although the experiments and analysis of the paper are very insightful and persuasive, it is only verified in a relatively simple setting. There are experiments on CelebA, but the number of attributes and their possible values are still kind of small. One or two experiments considering more complex scenarios will strengthen the paper a lot. - The paper claims that the model usually has the capability to compose the learned concepts earlier than this behavior shows up. How will this finding influence our understanding of (or provide insights to improve) the practical systems? Does this correlate to early stopping (the switching point from underfitting to overfitting)?

Questions

- The legends in Figure 2-bc (so as other learning dynamics figures) are a bit hard to read. - In Figures 2 and 3, the learning curves exhibit a “zigzag” pattern, i.e., first pointing to the training example sharing the strongest concept value, and then converging to the memorization one or comp-gen one. A similar phenomenon is also observed in a general classification problem discussed in [1]. Is there any similar reason behind these two findings? - Both the concept signal and sigmoid function are denoted by $\sigma$, which is confusing. - What are (0.4, 0.4, 0.6) to (0.3, 0.3, 0.7) in line 216 means, normalized RGB values? - The color bar of Figure 6 might have the wrong legend (it is the percentage of masked $\alpha$, but the bar ranges from 1 to 3.5). - The paper concludes that (in line 300) concept learning is a well-controlled phase transition, but the observed behavior can be arbitrarily delayed. Are there any methods that can reduce this delay? - [2] studies a very similar compositional generalization problem (but in a representation learning setting). It would be helpful to also discuss that in the related work. The discussion of simplicity bias and Kolmogorov complexity in [2] might also be a potential explanation for the observations presented in this paper. [1] Ren, Yi, Shangmin Guo, and Danica J. Sutherland. "Better supervisory signals by observing learning paths." ICLR 2022 [2] Ren, Yi, et al. "Improving compositional generalization using iterated learning and simplicial embeddings." NeurIPS 2023

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The experimental setting is kind of simple. But I think it is sufficient for a phenomena explanation (and theory) paper.

Reviewer uzj67/10 · confidence 4/52024-07-08

Summary

This paper introduces a new framework, "Concept Space," designed to study the learning dynamics of capability and behavior in generative models. It uses the concept signal to measure the rate of concept learning. They trained a diffusion model on a simplified synthetic dataset to validate their hypotheses that transitions in model capability occur much faster than the corresponding changes in behavior. This work provides empirical insights into understanding and controlling the development of AI models' capability and behavior.

Strengths

1. This paper is well-structured and clearly articulated. 2. This paper introduces the Concept Space framework and verifies several interesting combinatorial optimization results on synthetic datasets. 3. The experiment setting is meaningful and interesting.

Weaknesses

1. The rationale for modifying the differences in attribute values to represent different levels of concept signals is unclear. How does this relate to the formula $σ_i=|\partial G(\vec{z})/(\partial z_i )|$ The derivation of equation (1) is also not clear. 2. Uncertainty in generalization: Despite the introduction of the Concept Space framework, there is still uncertainty about the model's ability to generalize to real-world data. While concept learning is easily validated on synthetic datasets, it is unclear how to effectively validate this approach on large-scale datasets and large models. Can you provide some insights?

Questions

N/A

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Reviewer G81k5/10 · confidence 3/52024-07-11

Summary

This manuscript investigates the distinction between capabilities and behaviors in diffusion models. Using a toy task, the authors analyze at what point during training the model begins generating objects with correct specific features (in particular color and shape). They find that increasing the salience of certain attributes (“concept signal”) makes learning of those attributes faster and further causes the model's compositional generalization to collapse onto the nearest training point after intermediate training times. They replicate qualitative features of these learning curves with a mathematical toy model. They then use different interventions that enable the model to generalize compositionally much earlier than they would otherwise --- the fact that these interventions all yield generalization at the same time suggests a specific time point where the model has learned the underlying capabilities without this necessarily manifesting in behavior. Finally, they use this toy setup to analyze the impact of underspecification on compositional generalization.

Strengths

1. The topic of compositional generalization and concept composition in generative models is important and generally still poorly understood. The authors did a good job motivating this and its connection to the distinction between capability and behavior/competence and performance. 2. The findings in section 4.4 are surprising and notable and the authors replicated this behavior across different model seeds and methodologies (though I have a couple of questions, see weaknesses, point 3). 3. The toy model for underspecification provided helpful intuition and appears to be a particularly simple task that gives rise to such underspecification. 4. The manuscript is generally well-written and the figures are generally easy to follow. 5. Concept memorization is a useful phenomenological finding.

Weaknesses

As noted above, I think there are several interesting findings in this manuscript. However, in its current stage, I believe that it falls short of its stated goals. More specifically: 1. As far as I can tell, sections 4.1-4.3 and section 4.4 get at different forms of "capability." Specifically, the concept space studied in sections 4.1-4.3 distinguishes whether the model is able to generate certain properties of the image (e.g. color and size). In contrast, section 4.4 demonstrates that certain interventions can improve the models accuracy on the compositional task substantially. Since those interventions don't fundamentally change the model's capabilities, this suggests that the model has already learned to do the right thing and these interventions simply surface that capability. It's unclear to me whether that is necessarily apparent from the concept space behavior. Put differently, the model, in principle, could still be performing extremely badly according to the concept space but have already exhibited this transition. (You seem to be getting at a related point in section 6, "Why concept space?" and the supplementary figure, but it's unclear to me what the time of concept of acquisition is. To the extent that it relates the findings in section 4.4 to the findings in the previous sections, I think you'd have to show this across different models rather than using a single example of a model.) 2. I did not understand the role played by equation (1). First, the defined energy landscape always has its minimum (for high t) at $c_1,c_2=1$, so why can these curves tend to different corners of the concept space? Second, I did not understand how this differential equation is grounded in model behavior, as it does not appear to be a simplified learning model. Rather, the goal seems to be to recapitulate the (rough) trajectories in concept space, so I'm not sure what insights are gained from that. In particular, the fact that the model first tends towards the correlated training point (e.g. the small red circle) during learning is built in by the definition of the sigmoidal function. 3. I think the related work section on compositional generalization should provide a better overview of existing insights into the questions you're asking. Right now, you're only citing a number of papers investigating these questions, but I think it would be important to actually give an overview of what these papers are presenting and investigating and how it relates to your own findings. In particular, it's unclear to me what was previously known about the impact of underspecification on compositional generalization. 4. I think it would be important to report standard errors or some sense of deviation across different model seeds. It is still important in my opinion to understand the reliability of these qualitative findings --- e.g. how similar are the concept space curves you're presenting across different initializations? All in all, I think the paper presents several interesting findings, but, in its current state, leaves unclear how these findings fit together. On the one hand, the concept memorization finding is interesting and works well together with the underspecification finding. I think for both of those findings it would be important to more thoroughly evaluate how reliably the model actually ends up generalizing compositionally (e.g. across different seeds of initialization). In addition, it would also be helpful to give additional mathematical insights (or give an intuition in a different way) into why the observed behaviors emerge, as I don't see the current mathematical model as helpful on this end. On the other hand, the finding in section 4.4 is also interesting, but it remains unclear how it is related to the concept space framework and, if it can be explained in terms of learning both of these capabilities, why the presented intervention mechanism can help the model generalize compositionally.

Questions

1. Could you clarify how sections 4.1-4.3 and section 4.4 are related (see weaknesses, point 1)? 2. Could you clarify how you determined equations (1) and (2) and what we can learn from them (see weaknesses, point 2)? 3. As noted, I think the findings in section 4.4 are really intriguing, so I want to make sure I understand exactly what is going on there. a) Since you're using a binary classifier to assess performance, are the overprompted colors/sizes really identical to the original colors/sizes or is it possible that e.g. the color produced from overprompting is different from the ground-truth color and just more clearly on the correct red/blue side of the hyperplane? It would be useful to see a few examples of generated images here. b) It seems that you're only using one model seed in Figs. 4(c) and (d). For 4(d) you explained that you only used the one with "full capability" (I assume that's the one with an accuracy close to 1.0?). Why did you only use one model for Fig. 4(c) (or are these multiple lines that are just strongly overlapping?)? c) I think the fact that there's only one model that reaches full compositional generalization accuracy qualifies these findings a bit, in particular as it means that the latter methods only present a single sample. Would the other models improve in their performance as well if they were trained for longer? d) I would suggest providing an additional figure where you plot the different curves for each model seed on top of each other, as it is currently a bit difficult to judge how precisely the transition times overlap. 4. I'm not very familiar with the literature on underspecification --- have other papers previously noted its effect on compositional generalization (i.e. the "strawberry"/"yellow strawberry" effect)? If not, I think it'd be worth emphasizing that a bit more --- if yes, it would good to emphasize that as well. **A couple of minor notes** L. 29: “pre-training on such models” -> “pre-training of such models” L. 32: What is the “model experimental systems approach”? L. 69: Space before citations missing. L. 96: Is $S$ just the support of the probability distribution? L. 99: Is $F$ a stochastic function? L. 117-118: Isn’t $G^{-1}(Y)$ in the concept space, i.e. it should match $z$, not $h$? Figure 2: Why do the two trajectories for 01 and 10 illustrate concept memorization? I would have thought that this was illustrated by the trajectories for 11 that end up near 01. L. 247: What does it mean to mask the token? Set it to zero?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

As the authors acknowledge, they focus on toy synthetic data here. Furthermore, their analysis is largely empirical in nature, leaving unclear the exact reasons why the models generalize or don't generalize. I think the authors have adequately communicated the limitations of their work overall.

Reviewer cQgr5/10 · confidence 4/52024-07-12

Summary

The paper introduces "concept space", a framework for analyzing the learning dynamics of generative models, focused especially on compositional generalization. The key contributions are: - Introducing the concept of "concept signal" that governs the rate of concept learning (and in particular determine learning speed), and is the key driver of the generalization dynamics - Analyzing learning dynamics using the concept space framework on a synthetic dataset of 2D objects. - Claims about how concept signals shape the geometry of learning trajectories. - a proposal on interventional protocols to uncover hidden model capabilities. The paper uses a conditional diffusion model trained on synthetic data to study how concepts like shape, color, and size are learned and composed together. They show that concept signal levels determine learning speed and generalization dynamics, and that model capabilities often emerge before observable behaviors. The paper also explores how underspecification hinders compositional generalization.

Strengths

Originality: - Concept space and concept signals are a novel lens for analyzing the learning dynamics for diffusion models. - The claim that "capability consistently emerges before behavior" is interesting and well-backed, and is thus a novel insight into how these models behave over the course of the training. Quality: - Operating on the activation space and embedder patching seem like appropriate tests for the claim of testing . - It shows the intuitive result that model's generations for unseen combinations initially gravitate towards the training class with the strongest concept value. Clarity: - The claims are clearly presented with an explanation supported with figures. Significance: - The results suggest that the more distinguishable or salient a concept is in the training data (i.e., the stronger its concept signal), the faster the model will learn to recognize and generate that concept. If true for a broader class of generative models, the result could be widely applicable in data design choice for models.

Weaknesses

Significance: - Overall the setup is too simplistic to have the claims be transferable to a broader set of generative models or even diffusion models. All the experiments use a simplified synthetic dataset with 2D objects and binary concept variables; while good for control, it limits the complexity of the kind of concepts real-world models learn by a massive amount. Real world models are learning more complex, hierarchical, and interdependent concepts, not simply characterizable by things like size or color, and use continuous concept variables rather than binary. - The concept signal relies on knowing G, the data generating process. In real-world models, the true data generating process is often unknown, concepts might not have clear, differentiable manifestations in the input space. - The toy model for learning dynamics is based on a simple energy function with sigmoid activation; it's not clear why "concept signals" should be able to capture any non-linear interactions between concepts. Presentation of the claims: - The authors use the word "capability" when they really mean the model's internal ability to understand and compose concepts. The wording is quite unclear and should be clarified. - The graphs are pixelated, low-quality, poorly labeled and hard to understand. In particular, you should explain what the colors and the axis clearly mean in Figure 3, have multiple random seeds analyzed with error bars for overprompting and linear latent intervention in Figure 4, and make Figure 5 (b) higher quality. - Linear latent intervention assumes linear separability of concepts in the latent space, which breaks down in more complex settings than the synthetic data setting. - There should've been a higher focus on results on CelebA in the main sections of the paper.

Questions

- Would we expect overprompting to still work when concept manifestations are complex? - Doesn't the embedder patching technique assume that the final checkpoint has disentangled concepts? Why should we expect to be the case? - The paper briefly mentions experiments with the CelebA dataset in the appendix. Could the authors elaborate on how well the findings from synthetic data translate to this more complex dataset?

Rating

5

Confidence

4

Soundness

3

Presentation

1

Contribution

2

Limitations

- The paper should've compared the concept space framework to other existing methods for representations in learning dynamics (e.g., basic linear probing techniques, representational similarity analysis) as baselines for our understanding to see how much "concept space" improves it. - The models are fairly simple and small, and the study should further look into whether any of the techniques (e.g. overprompting, linear latent intervention etc.) still continue to hold in setting that more closely resemble real-world conditional diffusion models.

Reviewer G81k2024-08-08

Many thanks to the authors for their careful rebuttal, which has addressed many of my concerns. In particular, I appreciate the authors adding a quantification of uncertainty to their figures and I will increase my score to 5.

Reviewer o8es2024-08-08

Thanks very much for the author's hard work. The new results and discussions indeed strengthen the paper. I would like to increase my evaluation to 8. I am looking forward to seeing the final version of the paper.

Reviewer uzj62024-08-09

Thank you for the detailed response. I really appreciate the idea and motivation behind this paper. The rebuttal has significantly improved the quality of the paper, and I now feel more confident in recommending its acceptance. As a result, I have raised both my score and confidence level.

Program Chairsdecision2024-09-25

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC