Why Generative AI Helps Some People More Than Others

Fourteen empirical studies, and four dimensions for comparing them

By 無擷

October 5, 2026 · v4.2 (October 6, 2026: citation and reference numbering and corresponding links only)

A professional-writing study published in 2023 asked a few hundred working professionals to complete two short writing tasks online, each twenty to thirty minutes long. Everyone wrote the first one unaided. For the second, a randomly chosen half were allowed to use ChatGPT. The results were tidy. People with the AI took about 40 percent less time, their work was graded 18 percent higher, and the people who had scored lowest on the first task improved the most on the second. [1]

Another study followed 5,172 customer-support agents at a single company as it rolled out, in waves, a generative-AI tool that suggested replies. Resolutions per hour rose 15 percent overall. For agents in the lowest skill quintile, the gain was 36 percent. The most experienced agents barely got faster, and their quality slipped slightly. [2]

Looking at just these two, it is easy to reach a simple conclusion: generative AI is a leveling technology. It lifts the people at the bottom and does little for the people at the top.

But consider two more studies. In Kenya, 640 small-business owners were randomly assigned either to receive a GPT-4 business advisor over WhatsApp for about five months or to go without. On average, there was no detectable benefit. Split the sample by baseline performance, though, and the picture splits too: for owners who were already doing well the point estimate was positive, and owners who were doing poorly got significantly worse. Both groups received similar advice; process data suggest the difference may have been in what they did with it. [3] In another study, 16 experienced open-source maintainers were allowed to use an early-2025 mix of AI coding tools on a randomly selected subset of their own tasks. They expected to finish 24 percent faster. They finished 19 percent slower. [4]

Same class of tool. Sometimes it narrows a gap, sometimes it widens one, sometimes it slows down experienced maintainers. Why?

This article is an attempt to answer that question. It is not a systematic review. It is a focused reading of fourteen empirical studies, each of which used real people and measured how well they performed a task. Thirteen of them report how the effect of AI differed by something measured before the AI arrived: experience, baseline performance, education, expert status, or an independent test. The fourteenth, the open-source maintainer study, reports only an average effect and is included because it marks the evidence boundary for agentic tools. To these I add a small set of papers about measurement. The aim is to put results that look contradictory into one comparative frame: who the person is, what the task is, how the person and the AI divide the work, and what outcome is being measured. These four dimensions are a way of organizing comparisons, not a causal model that has been identified. By the end you will see that these fourteen studies do not support any rule of the form "the weak always gain more" or "the strong always gain more." What they do support is a more useful set of statements about which specific studies, which outcomes, and which direction, along with an honest list of what remains a candidate explanation rather than a finding.

The studies that show gaps narrowing

Start with the studies that show gaps narrowing. They do not all measure the same thing.

Besides the writing study, two programming studies examined completion time and output counts. Researchers at GitHub recruited 95 developers, gave half of them the Copilot code-completion tool, and asked everyone to implement an HTTP server. Among those who finished, the Copilot group took 55.8 percent less time. Sorting by self-reported years of experience, the benefit shrank as experience grew, but that result was only suggestive, short of conventional significance, and the analysis covered only those who completed the task, 35 people per arm. [5] Later, some of the same researchers took part in three field experiments inside companies, covering 4,867 developers, and found that Copilot raised the number of completed tasks by about 26 percent. That figure is a local effect under instrumental-variables assumptions: it applies to the people whose use of the tool was changed by the random assignment, and it is neither the average effect among all users nor the average effect among everyone assigned. In the Microsoft subsample, junior developers' pull requests rose 40 percent and senior developers' rose 7 percent, which looks like a large difference, but the difference was not statistically significant. [6]

Now quality. An online experiment randomized 1,174 participants to work on a business problem with or without GPT-4.1. The performance gap between more- and less-educated participants, 0.548 standard deviations without AI, shrank to 0.139 with it, a reduction of roughly three quarters. [7] In another experiment, 293 writers each produced an eight-sentence short story; some were randomly given zero, one, or five story ideas generated by GPT-4, and 600 readers rated the results. With five ideas, novelty rose 8.1 percent and usefulness 9.0 percent. Sorted by an independent creativity test taken beforehand, the Divergent Association Task, writers with low scores gained 6 to 11 percent in novelty while high scorers gained less. The side effect was that AI-assisted stories became more similar to one another. [8] And in a large experiment with consultants at Boston Consulting Group, the inside-the-frontier cohort involved 385 consultants working on 18 consulting tasks designed by the researchers; those randomly assigned to the GPT-4 condition, compared with the control, completed 12.2 percent more tasks, finished faster, and produced higher-quality work. The working-paper version adds that consultants in the bottom half of the baseline distribution improved quality by 43 percent, those in the top half by 17 percent. [9]

Read together, these studies leave the impression that lower-baseline people gain more. But notice that they are looking at different things. In the programming studies, years of experience gave only a suggestive slope, and the junior–senior difference in the company data was not significant. The writing study sorted people by first-task score, the business-problem study by education, the short-fiction study by an independent creativity test, the consulting study by baseline half. A baseline score, a degree, and a creativity test are not the same variable, and each one's relationship to the gain holds only within its own study. What these studies do share is this: in each, the AI handed part of the output directly to the person, completed code, a generated draft, a list of ideas, a suggested reply, and the outcomes included time, output counts, or scores. Whether the tasks had "clear standards" is not something the studies operationalized in any common way, and this article does not label them.

Two cautions belong here.

The first is statistical. A finding that the lowest scorers improve the most carries a built-in suspicion. If the same yardstick measures both the baseline and the improvement, someone who happened to have a bad day at baseline will "improve" next time by regression to the mean alone. This is not hypothetical. A study of law students had 48 of them take a real, unaided final exam, then, after some training, answer old exam questions with GPT-4. Multiple-choice performance rose by an average of 29 percentile points, essay performance showed no clear improvement, and the lower the baseline, the bigger the gain. But there was no randomized control group, the baseline score was used both to rank students and to compute their gain, and the authors themselves note in a footnote that regression to the mean may be the main driver. [10] The study retains descriptive value; it cannot be used to show that AI harmed the top students. The writing study that opened this article is more convincing precisely because its randomized control arm gives a comparison at the same baseline position. Its working-paper version reports that, regressing second-task score on first-task score, the control group's slope was 0.49 and the AI group's 0.25, a difference with statistical support; with valid randomization the two groups are comparable on first-task scores, so the difference in slopes cannot be waved away as regression to the mean. To be clear, 0.49 and 0.25 are regression slopes, not correlation coefficients, and not a measure of how much the overall gap in scores closed. When you read a study of this kind, look for a control arm, a scale without a ceiling, and a stable baseline measurement.

The second caution is definitional. Speed is not quality. The customer-support study is one of the few that reports both, and for its most experienced agents speed was flat while quality dipped. The two programming studies are cited here for time and task counts, and those results on their own do not imply any improvement in quality. When a study reports that AI let novices catch up with experts, ask what, exactly, they caught up on.

The cases that run the other way

Back to Kenya. [3] What sets this study apart is the task: the owners were making real business decisions in their own businesses. The AI did nothing for them; it only advised. After five months, owners with high baseline performance improved by about 0.16 standard deviations, close to but short of conventional significance; owners with low baseline performance declined by 0.08 standard deviations, and that decline was significant. The difference between the two groups has the support of an interaction test. The researchers examined process data and follow-up text material and found that the two groups received similar advice. That suggests the difference may lie in implementation, but it is an explanation suggested by process evidence, not a demonstrated mechanism, and it does not mean that every high-baseline owner benefited. The outcome, it should be said, was a self-reported performance index, a boundary worth remembering.

A second study offers a cleaner contrast, because it randomized the way the AI participated. [11] The researchers recruited expert and non-expert advertising copywriters and assigned them to one of two collaboration modes: in one, a large language model ghostwrote the copy; in the other, it gave feedback on copy the person had written. The ads then ran, and real click-through rates were measured. Ghostwriting produced no significant benefit overall and actually hurt the experts, which the authors attribute to anchoring on the AI's draft. Feedback raised non-experts' click rates to near expert level. Same task, different collaboration arrangement, and the direction of the result differed between experts and non-experts. Among the fourteen studies, this is the one that randomized the "finished product versus feedback" condition within a single study, which is why it carries more weight than any comparison across studies. The material verified so far supports that directional comparison of modes; the specific interaction figures for experts and non-experts still await checking against the published full text.

The consulting experiment has a second half. [9] Besides the 385 consultants in the inside-the-frontier cohort, another 373 worked on a task the researchers judged to lie beyond the AI's capability frontier at the time. Those assigned to the GPT-4 condition were less likely to get it right; the working paper puts the gap at 19 percentage points. This is easily read as "AI helps you inside its frontier and hurts you outside it." The direction may well be right, but two things need noting. The inside and outside groups were different people, so this is not one person flipping sign across two tasks. And the "frontier" was placed by the researchers according to their judgment of the model's abilities; how the AI performed on that task on its own, and how that was assessed, is not something the verified material settles. What the result shows is that different tasks can produce different outcomes. It does not let you attribute the flip to the task dimension alone.

Finally, the open-source maintainers. [4] The design randomized at the task level: across 16 developers, 246 self-chosen tasks were split between "AI allowed" and "AI not allowed," with the tools being early-2025 Cursor Pro paired with Claude 3.5 and 3.7, used in a mix of chat, completion, and agent modes. On average, tasks took 19 percent longer, with a plausible range from about 2 to 39 percent. It is an easy study to over-read. It had 16 participants, all senior maintainers deeply familiar with their own repositories, and no low-experience arm, so it cannot show that AI drags on experienced people more than novices. It used one specific tool combination at one point in time, so it does not speak for all agent tools or for later versions. What it does show is that for these people, these repositories, and these tools, "AI will surely speed things up" is not a safe default, even when the developers themselves were sure it would.

Put these four together and several candidate explanations appear: whether the task is open enough that the person must screen and reshape the AI's output; whether the AI delivers a finished product or feedback; whether the outcome is a score, real behavior, or self-report; and which generation of tool is in play. But the status of these explanations must be stated plainly. They come from comparing across studies that differ simultaneously in population, task, tool, and period. Cross-study contrasts can propose candidates; they cannot prove that any single dimension causes the effect to flip. "Openness of the task" in particular is only a candidate awaiting a test, and this article does not sort the studies into open and closed by intuition. Of the four, only "product versus feedback" has within-study randomized support, from the copywriting study.

Assisted performance is not learning

There is a subtler flip: performance with the AI present versus performance after it is taken away.

The business-problem experiment mentioned earlier included an unassisted module. [7] In the AI-assisted part, the education gap shrank by three quarters. In the unassisted module, in the authors' words, "a sizable education gap re-emerges." In that study, the leveling during assistance happened mostly at the level of the task, not the person, at least as measured in the immediate follow-up.

A high-school mathematics experiment pushes this further. [12] About a thousand students were randomized by classroom into three groups: one practiced with a standard GPT-4, one with a guard-railed "tutor" version designed not to give answers directly, and one without AI. During practice, the two AI groups scored 48 percent and 127 percent higher than the control. On the subsequent unassisted exam, the standard-GPT group scored 17 percent lower than the control. For the tutor group, no clear difference from the control was detected. That "no clear difference" needs careful reading: it does not show that the tutor version was equivalent to the control or harmless, and it does not show that it produced a learning gain. It says only that this study did not detect a difference. Heterogeneity by prior GPA and similar characteristics found limited or no statistical support.

The third study is closest to an engineer's daily life. [13] Fifty-two developers learned a library none of them knew (Trio) and completed tasks with it; half had a GPT-4o chat sidebar. These were not beginners; more than half had seven or more years of coding experience. Task time did not differ significantly between groups, but the AI group scored markedly lower on an immediate quiz afterward, an effect of about 0.74 standard deviations. Stratified by years of experience, the control group outscored the AI group at every level, though this stratification is descriptive and no interaction test was run. The researchers also identified six usage patterns from the chat logs, some of which correlated with better quiz scores, but those patterns were found after the fact, and nothing in the design shows that any of them "preserves learning."

All three studies measure immediate performance after the AI is removed, not skill months later, and their conclusions should not be merged. But they point at the same thing: a team that completes tasks better with AI is not the same as a team whose people have become more capable. For a manager, those are two metrics to track separately.

Ghostwriting, feedback, and execution are not one kind of collaboration

By now "how the work is divided" has come up repeatedly and deserves its own treatment.

Across the fourteen studies, the AI participates in at least four ways. Completion: Copilot continuing what you type. Chat: you ask, it answers, and it can produce whole passages of output. Advice and feedback: the AI executes nothing and only offers opinions, as with the customer-support reply suggestions, the Kenyan business advisor, and the story ideas given to writers. And agentic execution: the AI edits and commits directly in a codebase. What a study allowed and what participants actually did are different questions, and this article has not catalogued actual usage behavior study by study.

Only two studies randomly compared collaboration arrangements within a single design: ghostwriting versus feedback in the copywriting study, and standard versus tutor in the mathematics study. Their tasks, populations, and outcomes differ, and they cannot be merged into a general conclusion that products harm and feedback helps. In the copywriting study, ghostwriting hurt experts and feedback helped non-experts. In the mathematics study, the standard version lowered unassisted exam scores, and for the tutor version no clear difference from the control was detected. What the two local designs show together is only this: the collaboration arrangement is itself worth treating as a condition to compare, rather than a detail of the tool. Within the set of studies covered here, these are the only two within-study comparisons of collaboration arrangements.

The evidence on agentic tools is the newest and the least mature. Besides the maintainer study, Microsoft observed tens of thousands of its own engineers adopting command-line agents (Claude Code and GitHub Copilot CLI) between January and April 2026. [14] Adopters merged 24 percent more pull requests, with a confidence interval of roughly 14.5 to 33.7 percent. By level and tenure the gains formed a C shape: larger for more junior individual contributors and for more senior managers, smaller in the middle. But this is an observational study. Adoption was voluntary, and the authors state plainly that the result is an association, not a causal effect. The interaction terms cannot distinguish a tool that suits certain people from roles whose tasks happen to suit agents. Engineers with under a year of tenure may still have been onboarding. And a pull request count is not value; more PRs can simply be smaller PRs. Another observational study makes the same point from a different angle: grouped by coding activity, the gain in coding activity from AI tools was largest in the least active quartile, but the gain decayed along the chain from writing code to advancing projects to shipping. [15] Activity is not ability and decay is not causation, but the distance between "wrote more code" and "delivered more value" is one every engineering manager recognizes.

Within this set of studies, the randomized evidence on agentic tools is the maintainer study's average effect. A randomized or quasi-randomized interaction test by a characteristic of the person measured in advance is something this reading has not confirmed. On who gains more under agentic tools, this set of studies does not yet offer heterogeneity evidence to rely on.

What these studies can and cannot show

This is the place to pause on a methodological point, because it governs the wording of every paragraph above.

When a study randomly assigns AI and, under appropriate analysis, finds a statistically supported interaction, what it can identify is that the average effect of AI differs between the low-baseline group and the high-baseline group. That depends on valid randomization and appropriate analysis; a difference between two subgroup means is not enough on its own. And it does not establish that having a low baseline causes a larger gain: baseline was not randomized, and people with low baselines may differ in other ways too. Still less does it tell you how much any particular person will gain; a difference in group averages is not a distribution of individual gains. That is why this article keeps saying "average effects differ across groups" rather than "experience determines the benefit."

Quasi-experiments require a further step back. The customer-support study rests on the assumption that the timing of the staggered rollout was unrelated to trends in the outcome; this article has not evaluated whether that assumption holds. The Microsoft agent study is an observational association in which adopters chose to adopt; on the strength of that association alone, the change in pull requests cannot be attributed to the tool. The law-school study has no concurrent, unassisted control group. Each of these carries information, but information of different kinds, and they cannot be lined up on one axis and ranked by size, especially since their units differ: percentile points, percentages, resolutions per hour, standard deviations. As an aside, a preregistered meta-analysis found that the pooled effect of generative AI on programming productivity is larger in controlled experiments than in company and open-source settings, but that concerns average effects and does not transfer to the question of who gains. [16]

Can we measure in advance who will benefit?

Suppose the key really is how a person screens, checks, and reshapes what the AI produces. The high-baseline Kenyan owners, the copywriting experts anchored by a ghostwritten draft, and the developers whose usage patterns went with better quiz scores all hint in that direction. The natural next question is whether that capacity can be measured before the person ever touches the AI.

First, three things need to be kept apart. One is domain or task ability: how well you can do the job without AI, which the studies above proxied with baseline scores, tenure, education, and expert status. Another is general AI knowledge: whether you know how these models work, where they tend to fail, and how to use them. The third is hands-on verification ability: faced with a piece of AI output, whether you can actually tell right from wrong, check it, and fix it. The three are conceptually distinguishable. A domain expert may know nothing about AI; someone who knows AI well may fail to spot an error in a specific task; and spotting the error depends on domain knowledge. Whether they are empirically separable, and whether each predicts benefit on its own, is not something the literature read for this article has tested.

For general AI knowledge, a candidate instrument exists. A study published in Computers and Education: Artificial Intelligence developed GLAT, the Generative AI Literacy Assessment Test: twenty knowledge-type multiple-choice items with good internal consistency in a sample of 355. [17] In a small criterion study of 83 people, participants first completed a task without AI and then with a chat tool built on GPT-4o and retrieval augmentation. Controlling for domain literacy and the no-AI baseline, GLAT scores had a small but significant association with assisted performance. That is useful, but it must be read correctly. It predicts how someone performs with AI; it does not estimate how much AI causally added. The two sessions came in a fixed order, so the difference between them mixes in practice effects and task differences, and nothing here is an interaction test under randomized assignment of AI. Nor does the reliability measured in 355 people carry over automatically to a sample of 83, and whether participants were barred from using AI while taking the test is not something the verified material settles.

GLAT has been used independently, though cross-sample evidence is still accumulating. A study of 531 undergraduates used it to predict students' judgments about whether AI use in hypothetical academic scenarios was ethical; the score on the fifteen items that remain after removing the five ethics items showed a small positive correlation with judgment accuracy, a Spearman coefficient of about 0.23. [18] The criterion there is judgment about hypothetical scenarios, not real behavior, and it has nothing to do with gains from AI. A master's thesis with 543 graduating seniors is available only as a repository abstract: it reports that, after screening out low-quality items, the score on the remaining items is unidimensional but its reliability falls short of customary thresholds. [19] That finding concerns the post-screening score in that study, not every version of GLAT; since the full text is unverified, this article does not use it to reject GLAT and does not cite any of its coefficients. In short, GLAT is a formally published candidate measure of general AI knowledge with independent applications, not an accepted standard, and not a measure of hands-on verification.

For hands-on verification, an adjacent paradigm comes from research on "appropriate reliance" in human–AI collaboration: the degree to which a person follows the AI when it is right and overrides it when it is wrong. One study of predictive AI, that is, classifiers, proposed two indicators: the share of cases where a person who was initially wrong switches to a correct AI, and the share where a person who was initially right holds firm against a wrong AI. Both are defined on the subset of items where the person formed an initial judgment that disagreed with the AI; neither is a general acceptance rate over all items. [20] In a generative setting, a 2026 study had 432 undergraduates solve Python problems; for 14 problems the AI's initial suggestion was preset, 8 correct and 6 wrong, and acceptance or rejection was coded by whether the final answer agreed with the suggestion. [21] One finding was that AI literacy and appropriate reliance were essentially uncorrelated. The more important point, though, is the measurement itself. "Accept" or "reject" coded from the final answer is an observable choice outcome. It does not show that the person went through a discrimination process; a correct answer can be a lucky one, and a rejection can mean the suggestion was not read at all. In the literature read for this article, no study has been confirmed that measures hands-on error detection as a pre-treatment variable and tests its relationship to quality gains under randomized AI assignment.

Effective correction: a candidate worth taking apart

Which brings us back to the intuition: is a person's ability to push back on an AI and correct it the core indicator of whether they can use AI well?

I think it is a candidate worth taking seriously, provided it is taken apart first. At least five things hide inside the word "correction." Willingness to challenge, regardless of whether the challenge is right. Discrimination, the ability to tell which outputs are wrong, which can only be measured on tasks with ground truth, and which is distinct from acting on it, since action also depends on cost and goals. Repair, the ability to fix the error without breaking what was right, measured as net error reduction rather than number of edits. Wrongful edits, unfounded changes to correct content, which belong as a separate negative indicator. And cost: how much time and how many conversational turns it took to reach the same correction. A person who argues often but discriminates poorly will keep introducing new errors; a person who discriminates well but cannot repair will point out problems without delivering correct results; on a single "did they correct the AI" score, the two could tie.

Several boundaries follow. Correction is a two-party affair: whether the model goes along or holds its ground affects the result, so "the AI accepted my objection" is not evidence that I was right. Success has to be judged against the task's ground truth, not the AI's demeanor. When the AI is right, the person can still verify it but should not change it without reason; verifying and editing need to be recorded separately. In open-ended tasks, "disagreement" is often a clarification of goals or preferences rather than a correction of fact, and the two must be kept apart. It should also be said that effective correction is not the whole of hands-on verification ability: independently checking real output into which no errors were planted counts as well, and measurement need not always rely on planted errors. Finally, and most important: verification behavior observed during a task may be a cause, or it may merely reflect task difficulty or ability the person already had, so it cannot be read backward into a stable pre-existing trait. Turning it into a pre-treatment variable would require a measurement independent of the task, plus evidence of stability, discriminant validity, and incremental prediction; a single internal-consistency coefficient is not enough. For now, it is a candidate variable at the level of the collaboration process, not a validated predictor of who will benefit.

Back to the opening question

Same class of tool: why does it sometimes level and sometimes amplify? After these fourteen studies, my answer is that this set of studies supports no single rule; but organizing the comparisons along four dimensions, the person, the task, the collaboration arrangement, and the outcome, turns the apparent contradictions into a set of findings that can each be checked on its own.

In the randomized studies of writing, a business problem, and short fiction, people with lower first-task scores, less education, or lower creativity-test scores showed larger immediate quality gains. In the customer-support and code-completion studies, people with less experience or lower skill showed larger speed gains, some of which were only suggestive or not significant. In the Kenyan business study the direction was the reverse: a positive point estimate for high-baseline owners and a significant decline for low-baseline ones. On collaboration arrangements there are two local within-study comparisons, suggesting that "product versus feedback" and "guardrails or not" deserve to be treated as conditions in their own right. On performance after the AI is removed, the education gap re-emerged in the business-problem study, and the standard-GPT group in the mathematics study and the AI group in the new-library study scored lower unassisted; these are results of specific studies, not grounds for a general claim that gaps always return once the AI is gone. Under agentic tools, the randomized evidence available is the maintainer study's average effect, alongside an observational association inside one company; randomized heterogeneity by pre-measured characteristics is something this reading has not confirmed.

For a technically literate manager or research lead, these results translate into a few direct questions. Are you measuring speed, quality, or the person's capability? Is your tool completing, ghostwriting, advising, or executing? Does your task have a clear right answer, or does it require the person to screen? When you see "novices catching up with experts," did they catch up on a score or on a real outcome? None of these has a universal answer, but each has specific studies to consult.

For researchers, the clearest gap is this: in the literature read for this article, no study has been confirmed that measures, in advance and independently of the task, who can effectively screen and correct what an AI produces, and tests that measure against gains under randomized AI assignment. The existing local findings remain informative; using them to predict how a particular person will benefit still requires more direct measurement and validation.

A note on scope

This is a narrative, non-systematic review. The core studies were selected by four criteria: human participants; a measured task performance; a generative language-model tool as the AI; and reported subgroup or interaction results by a characteristic of the person measured in advance. One randomized study that reports only an average effect was added because it defines the evidence boundary for agentic tools [4]. Fourteen studies do not represent the whole literature, and the selection criterion "reported a pre-treatment difference" is itself a source of bias. The search was mainly of English-language literature; the core search closed on October 3, 2026, and the supplementary check of measurement literature on October 4, 2026. For some studies only the abstract or a working-paper version could be verified, as noted in the text and references; figures are taken from the latest accessible version. Wherever this article says "not confirmed" or "the verified material does not settle," it refers to the material verified within this scope, not to a claim about the original studies or the field.