A Pattern Language for Machine Learning Tasks

We thank you for your thoughtful feedback, particularly regarding the paper's positioning for the ML community. While we agree that better connecting to ML terminology would improve accessibility, we believe the substantive benefits for ML practitioners are stronger than your review suggests, and we welcome the opportunity to clarify this. Before we address your specific questions, your key concerns appear to be: 1. How our framework differs from standard regularization/multitask learning 2. Benefits for ML practitioners beyond conceptual understanding Regarding the distinction from regularization/multitask learning: As you observe, tasks can express both of these (modulo choices of stat. divs. and weighting), but tasks can additionally express "multi-learner multi-task" learning; rather than simply adding loss terms to a single model, our framework guides the principled construction of modular objectives for distinct learners, which work together to solve problems. While that in itself is not new -- notably GANs work this way, but the discriminator is 'thrown away' at the end of training -- we seek to capture the phenomenon in generality. As demonstrated in Section 3.1 (ln 276), this perspective yields concrete benefits - we obtain a generative model that: - Works without supervised pairs and deterministically (cf. VAEs) - Trains stably without adversarial objectives (cf. GANs) - Allows targeted architectural improvements while preserving formally specified behaviours (Section 3.3's linear put) Concerning benefits beyond conceptual understanding: Theoretically, we have Proposition 2.10 which formally relates energy minimisation (and therefore, unsupervised learning approaches such as k-means) to autoencoding, and proposition 3.4 which relates the behaviour of strong manipulation to bayesian inversion. The proofs of these propositions in Appendix B is diagrammatic, making full use of the soundness and completeness of such diagrams for cartesian monoidal categories; if nothing else, it is a different way to do mathematics. Practically, echoing our responses to reviewer 2Po6 and mqHa, the benefits are downstream of a systematic recipe for developing new ML training objectives, which we will state explicitly in the paper if you find the perspective helpful: 1. Begin with an equational specification of desired behaviour - formal constraints that captures how components should interact. While there's no universal recipe for converting desired behaviors into equations, we provide a foundation of well-understood patterns (Section 2) that can be analyzed and composed to derive novel objectives - as demonstrated by our manipulation task. As novel examples, Section 3.1 specifies what properties a well-behaved attribute editor should satisfy, and Appendix E specifies how to translate algebraic conditions governing push and pop into a neural stack. 2. Replace abstract functions with learnable components (e.g., neural networks). The framework is agnostic to specific architectural choices, treating learners as universal approximators (detailed in Appendix A.2). 3. Transform equations into trainable objectives by choosing appropriate statistical divergences and combining weights (subject to hyperparameter search). These choices determine the specific overall loss function to be minimised. The resulting trained components will exhibit the specified behavior (in the computational limit) because the learning process finds smooth function approximations satisfying the constraints. As we then demonstrate, not only does this recipe work, the practical advantages we win are: 1. Systematic design of objectives (Section 3) 2. Formal guarantees about learned behavior (Props 3.3, 3.4) 3. Principled architectural modifications (Section 3.3) 4. Novel training approaches (manipulation task) Given the concrete benefits demonstrated in the paper, we believe our contribution is stronger than just providing a clear mental picture. We welcome your thoughts on whether these clarifications address your concerns about practical value. (specific questions to follow)

We deeply appreciate your insightful review, which cuts to the heart of the framework's potential. Your questions about algorithm relationships and diagram transformations are particularly valuable, as they highlight areas where we can strengthen the presentation to better serve the broader ML community. You raise three crucial conceptual questions, which we would like to address before your specific questions: 1. How to formally relate different algorithms through the framework 2. Rules for diagram transformation and equivalence 3. Clear demonstration of comparative insights Regarding algorithm relationships: As you've recognized, our framework enables formal comparison of seemingly different approaches. For instance, Proposition 2.10 demonstrates that energy minimization and autoencoding are equivalent under certain conditions - a relationship that's intuitive but hard to prove without our formal machinery. Similarly, Proposition 3.3 reveals manipulation as a refinement of CycleGAN, explaining why it achieves similar capabilities through simpler training. We do not know whether there exists a "systematic" way to relate descriptions of tasks to behaviours in the way you ask, and we aren't claiming we have such a method, because ML is complicated! However, part of our motivation for providing patterns alongside our proof examples was to give an accessible basis for beginning to investigate how these equational constraints inform model behaviour more generally. Regarding rules for diagram transformation: these are detailed in Appendix A, but we agree these could be better highlighted. Key operations include: - Task refinement (Definition 2.9) - Specialization (Section A.2) - Equational reasoning (exemplified in proofs B.12, B.14) Regarding comparative insights: We do provide several concrete examples, though these could be more explicitly signposted. - Proposition 2.10's equivalence explains that sufficiently powerful autoencoders successfully tackle energy minimization tasks (i.e. unsupervised learning) under appropriate conditions. - Proposition 3.3's refinement relationship predicts that manipulation would achieve CycleGAN-like capabilities, which we then verified experimentally. - In Section 3.3, understanding the formal requirements of manipulation helped us recognize that put could be implemented using an autoencoder and a learnt vector addition, while preserving desired behaviors. Addressing your specific questions: 1. The compound function α combines individual task losses into a single objective. As defined in Definition 2.2 (ln 111-124), α is a differentiable function (ℝ≥0)×|Φl| → ℝ≥0 that combines the individual divergences from atomic tasks. In practice, this is typically a weighted sum of the losses, as demonstrated in Appendix D.1's implementation where specific weights are given for each task component (ln 1487). 2. Yes, domain and codomain refer to inputs (left) and outputs (right) of diagrams respectively, as defined formally in Definition 2.1 (ln 94-97). For example, in Pattern 2.3's classification task, the domain is the data-label pair space and the codomain is the label space. 3. Precisely - as introduced on ln 54-55, "system" refers to the learned behavior being optimized and "specification" to the target behavior. 4. Good catch, thank you! Yes, there should be a Y in the domain to match the distribution 𝒴. We will correct this in the diagram. 5. The text above ∶⇒ indicates which learners' parameters are being optimized in that task. For PutPut, we're optimizing get's parameters to ensure it behaves consistently with the composition of puts. An explicit definition was found wanting by other reviewers too, so we will incorporate some version of the following into the paper: the symbol ∶⇒ denotes an abstract objective function to be minimised between the LHS and RHS diagrams (the formal semantics of which are given in Appendix A.1). This abstract objective becomes concrete modulo a choice of statistical divergence and weighting function in the context of other (atomic) tasks. If this is a helpful or clarifying framing, we will seek to include this explicitly. 6. To head off a potential misunderstanding, strong manipulation is regular manipulation plus additional tasks, so in this case a specialisation is immediate. More generally however, yes; this sort of schematised reasoning is exemplified in the proofs relating energy minimisation and autoencoder in Appendix B.1. B.12 and B.14 are pure diagrammatic reasoning. We see the framework's primary value as providing a mathematical "lingua franca" for reasoning about behavioral relationships across ML approaches. But, it's not a recipe book for implementation choices, and we cannot automagically derive equivalences between tasks. The framework wants to become a kind of toolkit to understand why certain approaches work or fail, and to predict relationships between seemingly different methods. We look forward to further discussions!

We thank you for your thoughtful and detailed feedback. We particularly value your attention to formal clarity and practical ML concerns. We note that several of your questions point to areas where we can improve the presentation to better serve the ML community, which we address below. Before addressing your questions in order, we understand your crucial concerns to be: 1. How the formalism handles practical ML concerns 2. Need for precise mathematical definitions 3. Appropriateness for an ML venue 1. Regarding practical ML concerns, our framework explicitly accommodates architectural choices and variable-length data. As demonstrated in Section 3.3 (ln 380-387), practitioners can freely choose architectures - we showed how restricting put to be linear yielded interpretable latent spaces without compromising the formal guarantees. For variable-length data like text, we note that our framework operates at the level of abstract datatypes (detailed in Appendix A.1); concrete implementations handle variable-length data through standard approaches like padding or autoregressive processing, as exemplified in Appendix F.1's text experiments. 2. Concerning mathematical precision, we acknowledge that some notation (particularly ∶⇒) needed better introduction. The symbol ∶⇒ denotes an abstract objective function to be minimised between the LHS and RHS diagrams (the formal semantics of which are given in Appendix A.1). This abstract objective becomes concrete modulo a choice of statistical divergence and weighting function in the context of other (atomic) tasks. If this is a helpful or clarifying framing, we will seek to include this explicitly. 3. Regarding venue fit, while our framework draws from category theory, its primary contribution is to ML practice through: - Enabling systematic design of objective functions (Section 3) - Providing formal guarantees about learned behaviour (e.g. Props 3.3, 3.4) - Guiding architecture choices (Section 3.3's linear put) - Facilitating discovery of new training approaches (manipulation task) The core systematic recipe of the paper is *learning representations of equational theories*, which we will include explicitly in the paper if you find the perspective helpful: 1. Begin with an equational specification of desired behaviour - formal constraints that captures how components should interact. While there's no universal recipe for converting desired behaviors into equations, we provide a foundation of well-understood patterns (Section 2) that can be analyzed and composed to derive novel objectives - as demonstrated by our manipulation task. As novel examples, Section 3.1 specifies what properties a well-behaved attribute editor should satisfy, and Appendix E specifies how to translate algebraic conditions governing push and pop into a neural stack. 2. Replace abstract functions with learnable components (e.g., neural networks). The framework is agnostic to specific architectural choices, treating learners as universal approximators (detailed in Appendix A.2). 3. Transform equations into trainable objectives by choosing appropriate statistical divergences and combining weights (subject to hyperparameter search). These choices determine the specific overall loss function to be minimised. The resulting trained components will exhibit the specified behavior (in the computational limit) because the learning process finds smooth function approximations satisfying the constraints. Addressing specific questions: "How would the proposed language formalize size-varying data?" As noted above, our framework operates at the level of abstract datatypes (Appendix A.1). Implementation details for variable-length data are demonstrated in Appendix F.1's text experiments. "What were the objective functions used during training?" The complete objective functions with weights are detailed in Appendix D. For example, D.1 provides the full weighted combination of losses for the Spriteworld experiments (ln 1476-1484). "How were task weights estimated?" As detailed in Appendix D, weights were chosen through standard ML practice (hyperparameter search) to balance the different loss terms. We acknowledge this could be better explained. "What does the text above ∶⇒ mean?" The text indicates which learnable parameters are being optimised in that task, as explained in Definition 2.1 (ln 94-97). "What does ∶⇒ formally mean?" As noted above, ∶⇒ denotes an abstract objective function minimising some divergence between the LHS and RHS diagrams, to be made concrete by supplying a choice of divergence and weighting function. We appreciate your careful attention to formal precision. While maintaining mathematical rigor, we propose adding clearer introductions to notation and better signposting of definitions to improve accessibility for the ML community. We welcome further discussion on any aspects requiring clarification.

We thank you for your constructive and challenging feedback. We particularly appreciate your careful attention to practical concerns, which we agree are crucial for bridging theoretical frameworks to machine learning practice. However, we note that several of the concerns you raise are addressed in the paper and appendices, which we detail below. Before we address all of your questions in order, we understand your crucial concerns to be: A. How theory translates to concrete implementations B. Practical or conceptual advantages over standard approaches C. Justification for focusing on attribute editing given the framework's generality A. Concerning implementation, Appendix D provides comprehensive details of architectures, hyperparameters, and training procedures. We are also happy to answer any further methodological questions you have! While there are signposts to these appendices in each substantial Figure displaying results (ln 298, ln 375, ln 429), we propose adding a subsection to Section 3 explicitly showing how diagrammatic specifications translate to familiar ML objectives and architectures, and more explicitly directing the reader to the experimental details. B. Concerning conceptual advantages, it is true that regularisation and multi-task learning are expressible in terms of tasks, but in addition, we may express "multi-learner multi-task learning", which does not appear to be a named concept; in this sense 'task' is a unifying view. Concerning subsequent practical advantages of this perspective, our framework provides a systematic recipe for developing new ML training objectives, which we will include explicitly in the paper if you find the perspective helpful: 1. Begin with an equational specification of desired behaviour - formal constraints that captures how components should interact. While there's no universal recipe for converting desired behaviors into equations, we provide a foundation of well-understood patterns (Section 2) that can be analyzed and composed to derive novel objectives - as demonstrated by our manipulation task. As novel examples, Section 3.1 specifies what properties a well-behaved attribute editor should satisfy, and Appendix E specifies how to translate algebraic conditions governing push and pop into a neural stack. 2. Replace abstract functions with learnable components (e.g., neural networks). The framework is agnostic to specific architectural choices, treating learners as universal approximators (detailed in Appendix A.2). 3. Transform equations into trainable objectives by choosing appropriate statistical divergences and combining weights (subject to hyperparameter search). These choices determine the specific overall loss function to be minimised. The resulting trained components will exhibit the specified behavior (in the computational limit) because the learning process finds smooth function approximations satisfying the constraints. As detailed in our answer to "How does this framework generate new or better components/tasks?", this recipe does deliver practical advantages. C. Regarding the focus on attribute editing: manipulation serves as a compelling proof-of-concept precisely because it demonstrates deriving novel training approaches from formal specifications. Additionally, Appendix E shows broader applicability through "neural data structures", demonstrating how our pattern language guides implementation of differentiable versions of classical data structures. We elaborate further in the answer to "Why is attribute editing chosen as the only task?". (comment continues)