An engine not a camera: Measuring performative power of online search

Summary

The authors describe performative power, a pre-existing proposed measure of platform market power, and give an approach to measuring performative power using a browser extension. The browser extension perform random modifications to the search results page of results from target search engines, and measure click behavior. The modifications and user clicks are logged, providing sufficient statistics to compute a variant of the performative power metric. They deploy the extension to 85 users, resulting in about 57k clicks, and produce performative power calculations based on these interactions. The authors then provide some discussion of the results.

Strengths

I feel there are two main strengths to the paper 1. The authors make the case for measuring performative power using a browser extension 2. The authors created a browser extension to measure performative power. They appear to have been somewhat careful, discussing issues with hiding the DOM until rewriting is complete, discussing some latency numbers that are not bad (around 40ms, certainly enough to impact user behavior, but not enough to be wildly visible). They discuss privacy implications of their storage and logging of user search events, and have taken a reasonable approach with this, logging only intervention ids and click positions.

Weaknesses

W1: supporting ongoing work. I didn't see a reference to the source code for the browser extension, or any discussion of it being available for further work -- this seems like a surprising omission, though perhaps I didn't read the right section? W2: on novelty, I have a hard time characterizing exactly where the main contribution of the paper lies. The development of the performative power measure, and the argument for its relevance in regulation, comes from prior work. The understanding of the impact of position on click likelihood is also quite heavily studied, so the headline numbers the authors show have reasonable support in the literature already. The authors make a small modification to the definition to incorporate changes to the page beyond the organic results, but this does not seem to be the key point. The observation that PP can be computed from a browser extension seems fairly straightforward, not a centerpiece for the argument of novelty -- such approaches have been taken for modifying search results and measuring interactions in the past. The paper could arguably make the case for taking a straightforward idea (browser extension for performative power) and exploring the many thorny problems in designing and deploying this measurement, but my second primary concern below is that these issues remain largely outside the scope of the paper. Overall, I feel that there isn't a clear case to be made for the dimension of novelty. W3: platform power. The authors paint a picture of developing a causal understanding of the "power" present in the platform to differentially route user attention across resources. My concern is that some interventions are sustainable (ie, the platform could actually implement such a modification) while others are not. As an extreme and somewhat laughable example, consider the intervention that replaced every search result with a link to the CEO's gofundme account. This would change clicks dramatically, resulting in high performative power, but nobody would argue that the platforms could sustainably deploy this intervention. Instead, the goal is to consider power the platform has to alter the *ongoing* distribution of user attention to online resources, so making this measure robust requires significant attention to the issue of reasonable interventions. The related literature the authors cite (cascade, eye-tracking, MLIR) is quite careful in these areas. It is not clear that placing a pure navigational result at position 3 instead of position 1 is sustainable, and the issue is not so simple: users habituate to platform behavior, and will respond differently when the platform stops behaving as expected, both in determining which results to consider and hence where to click, and in bigger ways (ie, changing providers). Correspondingly, platforms themselves rely on user feedback, which would be adulterated by these types of interventions, with unclear implications. For a proposal intended to be a "blue print for how performative power can be used to guide digital market investigations," I think it's a significant omission not to consider these issues.

Questions

I'd love to hear the authors' thoughts on the three issues I raised in the weaknesses section above. Here are a few smaller questions also (I'll start numbering at 4, assuming the three points above are questions 1-3) 4. Could you discuss the fact that you consider only a single click, given that the intervention is likely to produce a higher number of clicks than the control (as satisfaction with the top result will be lower)? Likewise, the browser extension has the capability to consider more details of the interaction, such as immediate click-back from an unsatisfactory result page, followed by a click on a following page. I know it's biting off a lot to consider these types of interactions, but perhaps you could discuss a little. 5. Nit: line 112 has a typo in the cite of Chuklin et al 6. Could you clarify the triggering for arrangements a4-a6? Do they trigger only on results that have ads / shopping content? How is the accounted for in the analysis? Sorry, this may already be specified in the paper and I may have missed it. 7. Latency sounds pretty good, but probably worth referencing literature about the impact on search user behavior from additional latency of this magnitude 8. Study participants are trusted individuals solicited by personal outreach -- this suggests they will likely trust the researchers, tend to give the browser plugin the "benefit of the doubt," and so forth. Could you discuss the issues of conditioning that result from this choice of users?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

I feel that the limitations I discuss above should be covered in more detail by the authors. For ethics review, the authors indicate that this type of study does not require IRB approval at their institution, so I take this at face value.

Summary

This paper describes an online experiment seeking to measure how much power online search provides have in terms of impacting what content people consume. In short, the study attempts to measure the causal effect of small ranking rearrangements on click-rate for a population of web users. The authors present experimental data that can be used to estimate the expected impact that operators of search engines, recommender systems, and other ranking technologies might have on viewership of items when they make small changes to their rankings.

Strengths

In terms of originality, quality, and clarity: - Originality is high. The authors review (in reasonably terse fashion) a number of studies that have sought to understand the impact of ranking items in search on attention (i.e., clicks and visual attention that items receive). The study is grounded in this past work and notes its major contribution is to begin studying this effect experimentally. - Quality of the study is high overall. See some concerns / questions below, but overall I would consider this to be a very convincing set of results. - Clarity is very high throughout. Experimental details are crisply described. In terms of significance - Reasonable dataset (for this kind of study) with 57,000 "real" queries from 80 participants - Known caveat that getting this data without direct access to search operator datasets is prohibitively expensive, hence why this is novel.

Weaknesses

Two weaknesses (with respect to venue fit and potential of the current draft) stood out: First, a minor note: this kind of experimental work might be somewhat unusual at NeurIPS as it doesn't engage with the modelling side of ranking. Personally, I do not think this should be a blocking reason -- I think many in the community would like to see future works that incorporate this kind of experimental approach -- but it felt like an important piece of context to note. Second, some readers may concerned with perceived issues with ecological validity. To some extent these are unavoidable in any experimental study like this -- there will always be some set of ecological validity concerns, they just trade off with each other. - While this choice is reasonable, it may impact the kinds of queries used: "The study participants are trusted individuals of different age groups and backgrounds, recruited by reaching out personally or via email." - It might be helpful (esp. given the work may impact policy discussions) to know more about the domain / type of queries, but very reasonable privacy choice to avoid sharing any information about that. - Very minor: It's (by choice of technology companies) unclear if the kinds of perturbations studied here map to the kinds of a/b tests or experiments that are frequently rolled out by those companies. I don't think this is something the paper needs to address explicitly, but is also worth noting.

Questions

A few questions that might be worth addressing in the next version of the paper - Are there major constraints you would expect to face if trying to apply these methods to arbitrary types of other ranking platforms (feeds, matching/marketplaces platforms, etc.). - To what extent, if any, do you expect the specific algorithmic / modelling choices made by platforms to matter here? - Are there domains that might be hard to study, because e.g. the number of items is too high?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Limitations of the methods are reasonably discussed throughout.

Summary

The paper titled "An engine not a camera: Measuring performative power of online search" presents a study on the performative power of online search engines, specifically focusing on how they can influence web traffic through the arrangement of search results. The authors designed and executed an experiment to quantify the ability of search engines to steer user clicks by rearranging search results, a concept known as performative power. The main contributions of the paper are as follows: **Conceptual Framework**: The authors build upon the recent definition of performative power to develop a framework for understanding the influence of digital platforms, particularly search engines, on user behavior. **Experiment Design and Quantitative Findings:** The study involved tens of thousands of clicks collected over a five-month period from more than 80 participants, providing robust quantitative insights into the causal effects of search result arrangements. The paper reports significant quantitative findings, such as the observation that consistently downranking the first search result by one position causes an average reduction in clicks of 42% on Google search, and even more for Bing.

Strengths

**Originality:** The paper introduces a novel concept, the performative power of online search engines, which is a significant contribution to the field of digital platform policy and regulation. **Quality:** The analysis is thorough, with robust quantitative findings supported by extensive data collection over a five-month period from a diverse participant base. **Clarity:** The paper is well-structured, with clear definitions and explanations of key concepts such as performative power and the methodology used for the study. **Significance** The researchers presents that search engines are active "engines" that can significantly shape and influence the information landscape, with important sociopolitical implications.

Weaknesses

1. **Limited Sample Size and Diversity.** The paper could benefit from a larger and more diverse sample to ensure the results are representative of different user behaviors across various demographics. 2. **Lack of case studies**. While the paper provides clear quantitative results, additional qualitative analysis or case studies could help interpret why certain patterns emerge, offering deeper insights into user decision-making processes. 3. **Lack of discussion on interventions**. The researchers provide a compelling framework for measuring this performative power, but the paper does not extensively address potential countermeasures or interventions that could help mitigate the negative aspects of search engines' performative power.

Questions

See Weaknesses

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The author has already discussed the limitations of the paper.

Summary

The authors conduct a user study on the performative power of search engines, i.e., how much search engine providers can affect the information seen by the end user by tweaking the algorithmic ordering of results. In the specific context and assumptions formulated in this paper, this essentially amounts to measuring click-through rate differences across arrangements of search results. The authors ran a live RCT experiment by injecting different arrangements directly on the result page with the help of a browser extension of rheir design. Results show that there are notable differences in CTR and therefore commercial search engines have a large performative power.

Strengths

The paper is very well written, with all methodology and assumptions being laid out clearly. Moreover the experiment design and scale seem sufficient for the task at hand according to my understanding. The results are made more solid and generalizable thanks to the use of different providers. Also this work is more complete than some previous user studies of this kind and the performative power angle is original and relevant.

Weaknesses

I have two major concerns : - I am not sure this paper should appear in NeurIPS. It does contin any neural system, nor does it indicate how to create one from the results. While these results could in general be useful to an ML practitioner working with search engines, this could be said of many other research outcomes from different fields, that ypicaltly don't appear in NeurIPS proceedings. - The paper does not relate enough to previous work on position bias and other types of biases in searh results. While I believe new, updated user studies are always valuable, I think the authors should compare their results with those obtained by past studies.

Questions

- Do your results contradict or enrich previous estimations of position bias ? If so, what explains this ? - Do you think your method base on the extension is more reliable, or complementary to other methods used in past studies, e.g., eye tracking, result randomization, intervention harvesting,... ? - How do your results relate to other types of biases that have been identified, especially trust bias ?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

2

Limitations

Limitations are correctly adressed and very clearly laid out.

Response to rebuttal

Thanks for clarifying about the chrome extension -- this answers the question completely. Regarding the other questions, I think the authors have made reasonable points in terms of extensibility of the framework for other actions and measures. I think my remaining question centers around weakness #3. Let me lay out the concern here more fully, and the authors can perhaps respond with counter-arguments, point out things I'm missing, etc. The authors' rebuttal discusses that users will not habituate to the intervention because the intervention is randomized and reasonably infrequent. I don't think this is true. If navigational queries result in the correct answer at organic position #1 97% of the time, that's good -- users develop an expectation of the engine's behavior for that query class. Now, let's say that 12% of the time, an intervention moves the #1 result to #3, so the correct answer is now at #1 85% of the time. This is a massive change in the experience: the error rate grew by a factor of 5 from 3% to 15%. A competitor might find the top result 96% of the time. The intervention could be made less often to reduce this gap, but consider what is happening here: we are considering what level of intervention would not change the user's perception of value from the system, but immediately then measure the PP of the system under the assumption that the intervention is applied 100% of the time -- this clearly represents a different user experience, and the platform would resultingly occupy a different position in a new equilibrium. In the language of the original PP paper, performative power is known to go to zero as competition increases towards a state of perfect substitutability. The question of how much a search engine can adulterate its experience before it suffers competitive losses is therefore a question about the state of competition. My main concern here is that a careful handling of this question is required: just how much can the platform really modify its results before, over time, users begin to realize that a competitor provides better performance. This question seems quite tricky. So my concern is that the current paper is an empirical study of performative power in search ranking, but leaves out the attempt to tackle what appears to be perhaps the single most critical issue in understanding just how much power a platform has. Note further that the original paper considers this question of the PP of a platform with respect to viewers in a carefully-chosen setting: that of content recommendations of content hosted uniquely at the platform under study. There is no perfect substitute competitor in this setting. Web search, on the other hand, is fundamentally about providing access to a public database of content, so the issue of what competitors do is much more important. I wonder if the authors could engage a little on this and share thoughts.

Comments on related work

We are happy to do more polishing to clarify the relation to prior work. But let us be more explicit that our goal is not to design fairer or more optimal rankings, nor to debias click data. Thus, the methods you mentioned have a different focus and they are not directly related to our work. Instead, we are interested in measuring the performative effect of a very specific ranking update; an update that was documented in the context of the Google Shopping case, and found to be anti-competitive. And we tackle the question of how to design experiments and gather data that allow us to obtain a plausible lower bound on the effect of such a specific update. What makes this challenging is that we are not in the position of the platform, but someone outside aiming to monitor a system. Intervention harvesting as we understand it is a method to build on implicit interventions and extract information from click logs of multiple historic rankers to debias click models, e.g., for better propensity estimation. But debiasing click data is not what we are looking for, as we are not interested in learning a ranking model from data. But we are interested in measuring outcomes for a specific intervention where we do not have logs available. Thus, our focus is to emulate such an intervention and gather click data to estimate its effect. This is unrelated to methods that build on top of available logs to design (fair) ranking algorithms, such as [1,2]. In general, an important distinction to most existing literature in ranking is that we take the perspective of a regulator, not the platform controlling and optimizing their own algorithm. This leads to different problem statements and challenges. For example, we aim to establish a plausible use case specific lower bound for the effect of a specific ranking update performed by a particular platform. This requires us to gather strong empirical evidence from data collected on that platform rather than relying on modeling assumptions of how users might respond to such ranking updates, because such assumptions are necessarily inexact and hard to justify as quantitative evidence, even though at the same time they can be useful for learning, or for guiding the design of potential remedies. We hope this better answers your question. If you see a relevant connection we missed, after clarifying the specific problem we aim to solve, please let us know, and we are happy to discuss it.