FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learning

Summary

This paper considers preference-guided multi-objective learning from the lens of constrained vector optimization. Cone captures relative preferences. Constraints capture absolute preferences. Under boundedness and smoothness assumptions on the objective function, the paper provides gradient-based methods for the identification of Pareto optimal points. The paper provides convergence guarantees for deterministic and stochastic versions of various gradient-based algorithms and illustrates their applicability in a wide range of vector optimization problems.

Strengths

1) The proposed adaptive iterative update scheme has some nice properties, such as not requiring the initial model and $\theta_t$ to be always feasible, and not requiring different subprograms or different treatment of active inequalities. 2) Models both relative preferences and absolute preferences. 3) Proposed iterative methods come with convergence guarantees.

Weaknesses

1) This paper formalizes preference-guided multi-objective learning as a constrained vector optimization problem; however, the vector optimization aspect of the paper is not mentioned in the introduction section. It is not clear what “maximize” means, and it is not clear with respect to what cone the authors maximize. The introduction section gives the impression that the authors consider a multi-objective optimization problem, not a vector optimization problem. 2) Since there can be many solutions in the Pareto set, the set of solutions $\theta$ of PMOL may be very large. An important thing that is not mentioned explicitly in this work is how much of the true Pareto front ${\cal F}$ or the true Pareto set $\\{\theta : F(\theta) \in {\cal F} \\}$ the proposed algorithms capture when they converge. The theoretical results point to the convergence but do not point to what set of solutions the algorithms converge towards. Indeed, many algorithms whose task is to identify a Pareto set of solutions (either for multi-objective or vector optimization) come with guarantees on the returned Pareto set in the form of $(\epsilon,\delta)$-PACness results. Some success conditions that quantify the quality of the returned Pareto set or Pareto front are given in the following works. Auer, Peter, et al. "Pareto front identification from stochastic bandit feedback." International Conference on Artificial Intelligence and Statistics. PMLR, 2016. Ararat, Cagin, and Cem Tekin. "Vector optimization with stochastic bandit feedback." International Conference on Artificial Intelligence and Statistics. PMLR, 2023. The contribution of this work will be much clearer if the authors can demonstrate guarantees for the solutions returned by their algorithms, and compare them with the existing $(\epsilon,\delta)$-PAC guarantees in the related works. 3) Problem setup and preliminaries: “We first introduce new optimality definitions for PMOL that go beyond standard definitions of Pareto optimality”. These are some standard textbook results, not new. See and cite some of the following works. Boyd, Stephen P., and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004. Jahn, Johannes, ed. Vector optimization. Berlin: Springer, 2009. Löhne, Andreas. Vector optimization with infimum and supremum. Springer Science & Business Media, 2011.

Questions

1) Section 3.2 is confusing. As far as I understand, the goal of the paper is to optimize the objective function using gradient-based methods for a given ordering cone. First, a method that generates polyhedral cone matrix A from extreme rays of the cone is presented. This is okay, as it is just another way to express the cone. However, the second point, i.e., choosing $C_A$ for controlled ascent, is ambiguous. If the cone encodes the preferences of the decision maker, why do the authors need to change it by adding a new extreme ray? 2) Where are the green dots in Fig. 2 (a)-(e)? 3) What is the difference between the algorithms in Fig. 2 (e) and (f)? 4) What is the difference between the algorithms in Fig. 3 (c) and (f)?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

Limitations are adequately discussed.

Summary

This paper introduces a new algorithm called FERERO, which can handle both relative and absolute preferences in preference-guided multi-objective learning. The problem is formulated as a constrained vector optimization problem. The goal is to find the (approximated) $C_A$-optimal set that satisfies the absolute preference constraints. The cone $C_A$ can be interpreted as a relative preference that defines the objectives’ improvement directions. In each iteration, the FERERO finds a direction $d^{\star}(\theta)=-\nabla F(\theta)A_{ag}^T\lambda^{\star}$, derived by solving a subprogram. The $\lambda^{\star}$ in $d^{\star}(\theta)$ is derived from the Lagrangian of subprogram. Moreover, the approximated update rules and a stochastic version of FERERO are proposed for large-scale learning problems. The paper includes theoretical analysis of FERERO and its stochastic, approximated variants. The experimental comparisons with existing multi-gradient descent methods are also conducted to demonstrate the effectiveness of this approach.

Strengths

1. FERERO deals with both preference constraints and objective values in an adaptive way, eliminating the need to solve different subprograms at different stages. 2. The finite-time convergence rates of FERERO and its variants are established and the proof is solid. 3. The approximated rules, practical choice of preferences, and stochastic variants of FERERO-SA can be efficient for real-world problems.

Weaknesses

1. The step sizes in some of the convergence analyses, e.g., Theorem 2 and Theorem 3, depend on the $T$, which can be unknown in real-world problems. 2. The computational cost per-iteration can be higher than the traditional scalarization methods. 3. The studied problem is not well motivated. Compared to the traditional multi-objective optimization, the authors consider a more complicated case, where each objective function is a linear combination of the original multiple objectives. More real-world applications should be given to validate the significance of the problem. 4. C_A is not used in the experiments. That is, traditional multi-objective optimization is considered. This seems to be insufficient.

Questions

1. The preference constraints are expressed as linear functions of $F(\theta)$. Is this kind of linear structure assumption for preference constraints common in real world problems? 2. Could you explain your choice of step sizes in the experiment and how can it be related to your theoretical results.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper introduces FERERO, a novel framework for preference-guided multi-objective learning by depicting the task as a constrained single-objective optimization problem. It incorporates relative preferences, defined by a polyhedral cone, and absolute preferences, defined by linear constraints, into the above optimization problem. Both deterministic and stochastic algorithms with finite-time convergence guarantees are proposed. FERERO is validated through experiments on multiple benchmarks, demonstrating its effectiveness in finding competitive preference-guided Pareto optimal solutions.

Strengths

1. This paper is very well-written and its results are solid. 2. A novel concept called relative preference is proposed, which interests me. 3. A stochastic version is also proposed for fast gradient estimation. 4. Only one optimization problem needs to be solved for each preference, while some methods require solving multiple subproblems. 5. The experimental results are competitive and promising.

Weaknesses

I do not notice any obvious weakness. Some comments are given below: 1. What is the evidence of ``flexible``? Is flexibility important? 2. What is the meaning of ``cross`` in Table 1? Is it not mentioned in the literature or is it inherently impossible to do this? 3. What are the advantages of a method based on general partial order? Is it related to escaping weak Pareto optimality? 4. As far as I am concerned, Linear Scalarization and (Smooth) Tchebycheff can escape weak Pareto optimality. 5. The per-iteration may need more descriptions. 6. What if some absolute preferences are not available? e.g., $B_h$ or $B_g$ are not provided by the decision maker. Besides, the settings of preferences in the experiments are hard to find. 7. Are there some typos in the subtitles of Figure 2(e) and Figure 3(c)?

Questions

Refer to the above comments.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

The discussion is comprehensive in my view.

Summary

This paper tackled constrained multi-objective optimization problems with Pareto optimality generalized to partial order cone. The authors proposed an algorithm framework to iteratively solving a subproblem to get a update direction and aim at search for Pareto optimal solution in a certain Pareto front region associated pre-defined preferences. The paper presented rich convergence analysis for deterministic algorithm with exact subproblem solutions, deterministic algorithm with inexact subproblem solutions, as well as stochastic version where gradient is estimated through double samplings. The stochastic algorithm variant is tested on image classification and speech recognition tasks and compared against state-of-art benchmark algorithms.

Strengths

* This paper is well-structured, well-written and easy to follow. * The proposed algorithm framework can handle relative preferences where decision makers don’t need to specify explicit weights or threshold numbers. Also, the proposed algorithm has more flexible control on the search region on the Pareto front which is specified by extreme rays and/or constraints. This makes the algorithm more practical. * This paper provides comprehensive theoretical convergence analysis on the proposed three algorithm variants in general non-convex cases. The ideal of normalizing search direction by function values is novel to me. * The experiment on image and speech recognition tasks demonstrates its applicability to large scale neural network learning objectives. The improvement over recognition error rate is significant.

Weaknesses

* The relative preference seems being handled by both partial order cone associated with the objective function and inequality constraints. It is kind of confusing and redundant to have both preference cone and constraints defined in the objective space in the optimization problem. See the questions part for more details. The experiment on multi-task learning seems only tackling constraints. * In real decision making application, computing the ordering cone by solving another LP makes the decision making process more complex and less doable. * The authors selected number of algorithm iterations to compare computational cost, which leads to apple to orange comparison. Better to adopt number of function evaluations, number of gradient evaluations, etc. as the metrics.

Questions

* As both the relative preference and constraints are defined in objective space and both equality and inequality constraints are defined as linear functions of objectives, would it possible to convert at least inequality constraints to preference defined by a cone? * Can we reduce Pareto dominance defined in a polyhedral cone to Pareto dominance defined in Real space $R^m_+$ by taking $AF(\theta)$ as objective vector? * From Theorem 1 to Theorem 2, the second part on the left of Equation (2.5) $G(\theta_t)$ no longer present in equation (3.2). How is the inequality violation converging to zero in the single loop approximate algorithm variant? * Equation (5.2): what is the rationale to have the representation learning loss for Chinese and English differing by an exact number. Would it be more reasonable to have the loss gap smaller than a threshold? * It is not clear in Table 2 what is Emotion loss without referring to Appendix. * Figure 2(a): “LS only finds extreme points on the PF with one objective minimized." How are the secularization weights selected? Is the weight selection related to extreme rays? * Figure 4 clarification * 4(c) LS accuracy is missing. 4(a) also missing red square. * As mentioned in the paper, orange preference vector is not attainable. In (d)-(f), the proposed algorithm tends to search for Pareto front region closer to the ideal Pareto minimizer. Is this constraint violation controllable? * Figure (e) green cross given by XWC-MGDA seems dominating most of other solutions. * Minor issues: * Notation consistency: Theorem 2 equation (3.2) $d_t$ → $d_t(\theta_t)$. * Line 279: figure 3d-3f should be 3a-3c. * Figure 3: caption (c) and (f) should be consistent.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A