Idiographic Personality Gaussian Process for Psychological Assessment

Summary

This paper presents a multi-output Gaussian process model for classification in the context of psychological studies. It's based on a linear model of co-regionalisation leveraging unit-level latent factors and RBF kernels to jointly model population and individual personality traits to tackle an old debate in psychometrics regarding their original nature. The authors leverage a variational formulation to derive a lower bound for the model evidence and optimise the individual-specific, population, and GP-related parameters. An extensive simulation and real-data study is proposed, presenting longitudinal and multi-subject survey analyses, personality correlations and predictions.

Strengths

This paper is remarkably well-written and presented. The flow of derivations is easy to follow, and the illustrations are helpful and of excellent quality. The treated problem is far from trivial, with data that present several degrees of correlations (time, individual, population) and technical constraints (missing and categorical data). The experiments are impressive, with extensive comparison against many competitors and excellent results overall. The method shows great promise in providing practical insight into the domain, and even though I'm not a specialist in psychology, the ability to reconcile idiographic and nomothetic approaches seems particularly valuable.

Weaknesses

One could argue that the methodological novelty is limited as the presented multi-output GP model is fairly well-known in the GP community. However, I think its similarities and differences with GPLVM and GPDM are well presented, and the overall application is far from trivial regarding the nature of data (categorical outputs, latent variables, missing data, ...). This relative weakness is more than compensated by the strength of results and the potential such a methodology brings in a field like psychology, where the signal-on-noise ratio is generally low, and the nature of measurements is often challenging.

Questions

I only have a few questions regarding computation times, scaling and robustness of the inter-output covariance matrices, which are well-known limitations of co-regionalisation GP models in practice. - In this sense, could you provide numerical evidence and comparison regarding training and prediction times for IPGP and competitors? - Even though the number of tasks/topics remains probably limited in psychological applications, could you discuss a bit more, in Section 4.1 **Result**, the decrease in performance between full IPGP and the low-rank version. What could we expect for higher dimensions? Would the low-rank approximation remain robust for a more significant gap with the actual rank? - In my experience, several multi-output GP models (or maybe their implementations) are somewhat unstable when it comes to estimating $\textbf{K}_{task}$, and pathological cases can often arise. Have you experienced such problems? And if so, how did you manage to tackle this issue? **Typos:** Page 5: The indices of the CMD definition have a slight error. All terms are written as $R_1$, and the norm is 'f' instead of '$l_2$' Page 6: The title of 4.2 "Cross-secitonal" ==> Cross-sectional Page 7: "Both correlation matrices **displace** a block pattern" ==> display? Page 7: "questions corresponding negative emotionality" ==> corresponding to

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

In my opinion, the limitations are adequately discussed. The application spectrum and position in the related literature are well documented.

Summary

Gives a multi-task/output GP formulation for multiple time-series (or intrinsic co-regionalisation model) for pyschological assessments. Design of the factor loadings informs the task-correlations and reflects the knowledge of individual's correlation between his responses and inter-person correlations. More of an application paper rather than a paper advancing machine learning; but a good application nonetheless.

Strengths

1. The application is a natural fit for the model chosen, and results are good and convincing. 2. The use of Bayes factor for model testing make the paper better as an reference for this application. 3. Sections 4.2 and 4.3 evaluates the model from different aspects.

Weaknesses

1. The paper is weak from the perspective of advancing state-of-the-art machine learning algorithms. 2. The paper is totally unrelated to GPLVM and tasks that GPLVM are designed for. Mentions of GPLVM only confuses the reader. 3. Equation 3 needs fixing. 4. In section 3.2, the method is variational inference, and not stochastic variational inference. 5. Description of the setup in section 4.1 needs to be clearer to explain also in terms of $K_{task}$. 6. Line 192 mentions "informative prior". What is this "informative prior"? *Minor* 7. Line 319: "addressing" is too strong. Suggest to change to "contributing". 8. Line 321: "than" -> "over"

Questions

I do not have any critical questions.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

1. This paper introduces an innovative measurement framework utilizing the Gaussian process coregionalization model to resolve the question of whether psychological attributes such as personality exhibit a universal structure among the populace or are uniquely individualized. 2. An Idiographic Personality Gaussian Process (IPGP), a hybrid model that accounts for both the commonality of traits across people and individual-specific "idiographic" variations. 3. Gaussian process coregionalization model to interpret the responses from grouped survey batteries, adapted for non-Gaussian ordinal data, and employs stochastic variational inference for estimating latent factors. 4. The application of IPGP on both synthetic data and an original survey demonstrates its performance.

Strengths

1. The paper’s main innovation lies in its unique combination of multitask Gaussian process, which results in a conceptualization of the subject matter. 2. The innovative use of multitask Gaussian process in this paper offers a promising solution to a long-standing challenge of psychological assessment. 3. The quantitative and qualitative methods in this paper provide a potential in advancing psychological diagnosis and treatment. 4. The interdisciplinary nature of the research is interesting. 5. The authors provide a holistic view of the issue at hand.

Weaknesses

1. there are many MTGPs, the author ignored comparison with them. 2. incorrect description, such as: "multi-task structure is also known as the linear model of coregionalization (LMC)". 3. This represents a particular implementation of MTGP, but it doesn't introduce any novel methodological advancements. 4. The paper presents a valuable contribution, but its innovative aspects are limited, as it largely builds upon existing theories without introducing significant new insights. 5. The literature review appears somewhat limited. Expanding it to include more recent or relevant MTGPs could provide a more comprehensive context for the research. 6. The statistical analysis used in the study seems inadequate given the complexity of the data. 7. The research seems to be more of an incremental advancement rather than a novel contribution, which may limit its impact on the field. 8. The experimental evidence provided in the paper is not as comprehensive as it should be to support the claims made, suggesting a need for more extensive testing. 9. Only one experiment is related to psychological assessment. 10. No idiographic personality is discovered and presented in the paper.

Questions

Please see the weakness above

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Please see the weakness above

Summary

UPDATE: I am updating my scores in light of the excellent authors' response. This paper considers psychometric data composed of ordinal responses $y_{ijt}$, each of which represent how unit _i_ answered survey item _j_ during time period _t_. The key characteristic of this item-response data is that the same $N$ units are longitudinally surveyed on the same $J$ items over $T$ periods. The paper takes an ordered logit factor modeling approach to analyzing such data, wherein $f_j^{(i)}(t) = \mathbf{w}_j^\top \mathbf{x}_i(t)$ is the latent ideal point of unit _i_ on on item _i_ at time _t_, and $\mathbf{w}_j$ and $ \mathbf{x}_i(t)$ are the $K$-dimensional loadings and factor vectors, respectively. The paper moreover advocates an "idiographic approach [which] emphasizes _intrapersonal_ variation by requiring distinct loadings $\mathbf{w}_j^{(i)}$" which are different for each unit $i$. It is able to effectively achieve this by exploiting the repeated measurements of each unit-item $(i,j)$ pair over different time periods $t$. The proposed model is an instance of a multi-task Gaussian process (MTGP). Intrapersonal variation is modeled using a unit-specific kernel $K_{time}^{i}$. The full covariance is then $JT \times JT$, resulting from a Kronecker product of $\mathbf{K}_{\textrm{time}}^{(i)}$ (applied to input $T$ periods) with a low-rank matrix $J \times J$ matrix that represents covariance between survey items (or "tasks"). This all extends the previously-presented linear model of coregionalization (LMC), which was originally developed for the simpler case where observations of $(i,j)$ pairs are not repeated over time. The paper derives a variational inference inference algorithm for the model, which follows closely from previous work. The paper then reports three sets of experiments: 1) a synthetic study that reports excellent results on parameter recovery (when the true parameters are known), 2) a re-analysis study that purports to show that a non-dynamic (i.e., non-idiographic) version of proposed model is able to identify the correct factor structure (K=5) from survey data designed to assess the "Big Five" psychometric traits, and 3) an illustrative case study involving a novel longitudinal data set.

Strengths

The paper is very well-written. It brings an interesting application to life and convinces the reader that the proposed modeling approach is well-tailored to the problem at hand. The paper gives a clear review of the prerequisite concepts and presents its modeling approach clearly. The paper covers related work in psychometrics well is convincing that the modeling approach is novel within that applied community. Experiments bear out this claim, as the proposed model performs much better than models which are currently used in psychometrics. The paper presents a novel longitudinal survey dataset composed of $N=93$ subjects who were given personality assessment surveys over the course of three weeks. If I am reading the paper correctly, each subject was asked to complete a survey _six times per day_. This data sounds highly non-trivial to collect, and should be considered a main contribution in itself.

Weaknesses

The paper is vague about its technical contributions. It makes the following hedged novelty statement: "the first multi-task GP latent variables model _for dynamic idiographic assessment_". I took this to mean that the proposed approach is not very new technically, but it has never yet been applied to dynamic idiographic assessments. However, the paper also makes the general claim that it "advances the literatures on Gaussian process latent variable models...". I am of the opinion that tailoring existing modeling frameworks to new applications does provide a technical contribution, as it contributes a new view and set of interpretations/metaphors that can help to better understand the abstract model. I think the paper does contribute in this way. But I am unsure if the paper further contributes more substantially to the area of Gaussian process latent variable models, as the paper is vague about that. There is sloppiness in some of the main equations. For instance, equation 2 confuses a distribution with a random variable stating $p(\mathbf{f}^{(i}) \sim \cdots$, and I believe equation 3 confuses an inner with an outer product (shouldn't it be $w_i w_i^\top$?). There's also some confusing overloading of symbols between the background and the model sections. The background sets up the idea that an ideographic approach has distinct loadings $w_j^{(i)}$ different across $i$. But the proposed model itself does not seem to sport this as each $w_j$ is global for each task. I believe effectively the model does achieve an idiographic interpretation through the unit-specific kernel, but the connection between the background and model section is not clear. I think the second experiment has some major flaws. The paper claims this experiment validates the "Big Five" because model performance peaks at $K=5$. However, the model is only fit for $K=1...5$. We can clearly see from the log likelihood numbers that model performance for IPGP is monotonically increasing in $K$ for the values considered, suggesting that it will likely be even better at larger values of $K$; if that were true, it would not validate the "Big Five" theory. As this is an applied paper, I would expect a much higher level of rigor on one of the two applied case studies.

Questions

As this is an emergency review, after the reviewer discussion period, I am not sure if the authors will be able to reply. But if they are able, I would just ask that they respond to the various points made in previous sub-sections.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper did involve a fairly intense human subjects experiment for data collection, but the paper reports it was IRB-approved. Psychometrics of this kind are fairly common; I do not think the paper introduces any new potential negative societal impacts.