Proposal for the Future LLM Policy Sub-Team: Minimizing Reviewer Cognitive Overhead

community
1

Proposal for the Future LLM Policy Sub-Team: Minimizing Reviewer Cognitive Overhead

Acknowledging the Pragmatic Reality of Reviewer Burden

First, thank you to the maintainers, team leads, and moderators who put time into drafting and adopting the initial LLM policy for rust-lang/rust. Feedback from the community and maintainers highlights a few undeniable realities:

  • The current policy is a necessary mitigation, designed to protect maintainer bandwidth in an ecosystem where AI tools can exponentially increase PR volume at zero cost to the author.
  • Adding procedural requirements (such as mandatory test suites or extra documentation) can backfire, as verifying synthetic tests or generated prose often doubles the cognitive load on reviewers rather than reducing it.
  • Formal "accountability" is impractical in an anonymous open-source environment. Policy enforcement must rely on swift, practical triage rather than unenforceable expectations of individual liability.

With these constraints in mind, future iterations of the policy should focus entirely on reducing cognitive overhead for maintainers while keeping disclosure straightforward for well-intentioned contributors.

Key Focus Areas for Future Policy Iteration

1. Prioritizing Fast Triage Over Forensic Verification

Reviewers should never be expected to act as forensic detectors or engage in debates over whether a PR contains machine-generated code.

  • Zero-Friction Closure for Mechanical Threads: Reaffirm that PRs exhibiting unresponsive or mechanical behaviors (such as repeatedly copy-pasting reviewer feedback into an LLM and pasting the output back) can be closed immediately using a standard, non-punitive canned response.
  • Behavior Over Code Style: Emphasize that reviewers do not need to analyze code "style" for AI signatures. If a PR lacks necessary author understanding or places an undue burden on the reviewer, it can be closed based on technical review constraints alone, avoiding unproductive accusations.

2. Simplifying Disclosure for Good-Faith Contributors

The core guideline—“fine to use LLMs to answer questions, analyze, check, suggest, but not to create”—provides a clear general direction. To prevent confusion for honest contributors:

  • Lightweight PR Checkboxes: Keep disclosure simple within the standard PR template (e.g., a basic checkbox indicating whether LLM tooling was used for translation, linting, or drafting).
  • Clear Safe-Harbor Declarations: Explicitly reassure contributors that standard developer tooling (such as local editor completion or syntax-checking utilities) does not trigger additional review hurdles, encouraging honest disclosure without fear of immediate rejection.

3. Offloading Guidance to Mentoring Channels

Maintainer time is best spent on core engineering, not on moderating discussions around tool mechanics.

  • Automated Offloading: When a PR is closed under the LLM policy, maintainers can consistently point the author to dedicated community spaces (such as #llm-mentoring on Zulip), allowing the contributor to learn project expectations without consuming reviewer time.

Looking Ahead

The initial policy serves its purpose as a necessary defense mechanism to safeguard maintainer health. As a dedicated LLM policy sub-team is established, prioritizing low-friction triage workflows and maintainer-first defaults will ensure the governance framework remains sustainable for maintainers and clear for future contributors.

2

I don't think URLO is the right place to post this (nor do I believe the tone of your proposal is appropriate for the context). Maybe if/once an LLM policy team is established you can contact them about any potential changes to the policy.

Further links for people who are interested in Rust's LLM policy:

3

English is not my native language, and the choice of tone is based on the interpretability of the content. However, as you said, there will be a more formal space for discussion once the LLM group is established.

4

Properly verifying either of these things would require a review, thus doubling rather than lessening the load on reviewers.

Trying to hold anonymous internet users "accountable" or "liable" (whatever those may mean in this context) for their code is likely to cause further work and headache for maintainers, rather than less.

The status quo is not a solution to the problems, I can agree with that. If there was a solution, it would be worth a lot of money right now. In lieu of a solution, the teams have implemented mitigations, which have tradeoffs but are the best they can currently do.

7

The issue is not specific to Rust, but it is happening everywhere where software is being developed:

Now that AI has enabled people to produce lots of code with little effort, review becomes the bottleneck. If you insist that a human being carefully looks at every line of code changed, then you find that you lack the manpower to do all the reviews you want to do.

Short of rejecting most code without review, the only conceivable solution is to use AI to do most of the work of reviewing. Maybe a human approves or rejects based on a summary.

That idea, however, horrifies a lot of people. The main concerns brought forward are that code produced by AI is inferior in some way that no one can clearly define, or that is is necessary to cultivate a group of people who have an intimate understanding of the code.

Well, yes, if we want to insist that for every line of code there exist people who understand why it is the way it is, then that is a major bottleneck.

The social challenge of AI in creating code is similar to the role of excavators in digging large holes.

Before we had excavators, if you wanted to dig a large hole, people with shovels needed to do a lot of shoveling. You better had an army of people with strength and endurance and good shovels.

You can now argue against the use of excavators thus: if we use excavators, people will no longer be shoveling all day, and they will loose their strength and endurance, and they will forget where to get the shovels from. And there will always be some situations left where you want a person with a shovel.

And, yes, shoveling will never disappear entirely, but it is needed a lot less now that we use excavators.

8

This post belongs in https://internals.rust-lang.org/

9

Code produced by current LLMs tend to be more buggy than human-written code, and tend to lack the understanding of the larger surrounding context, and tend to be rather disorganized. As for the necessary to cultivate a group of people who understand the code, LLMs are limited by their context windows, which cause them to not be able to understand what happened in the past of the code.

10

Regarding the more buggy part: the only evidence I can find is a report that gets endlessly re-referenced which was published last December and looks at things even older.

Regarding the rest: I concede that it is a valid concern that users often don't give AI all the information it needs. Otherwise, in my experience, you get pretty well what you ask for.

11

Your tone was fine.

12

From a personal perspective, under certain methodologies or specifications, code written by LLMs is not an untouchable poison; one can configure multiple LLMs to review code according to predetermined methodologies or specifications and produce compliance reports, which can greatly reduce the workload of reviewers. The key issue is that Rust does not seem to have a clear, unified specification standard like Design by Contract, and this is the problem the community should face, rather than treating LLMs as a drug like Zig does and remaining stuck in the past.

13

I love comparing things with analogies. Your analogy almost got it.

This is true when you're producing mass product (holes to lay pipes, holes for cellars and foundations,...). You use automation, stop caring about the details, and what you get is gas explosions because of dug up gas pipelines, internet outages because of dug up network lines, injuries because of oversights. Still, all of that is somewhat handleable with insurance (which btw are all saying "hell no" to insuring the fact that you're using LLMs to do stuff).

But, we're not producing a mass product here. Not the 50473th cookie clicker game with abusive lootbox mechanics, not the 62428th badly written online shop with horrible filter behaviour, you get the gist

Open source is about having a handfull of specific curated projects that solve a problem and can be used and edited by anyone. My kingdom for 5 online shop softwares that are improved indefinitely instead of everyone thinking being on Amazon/eBay/.. is the height of selling things.

These projects don't need velocity. They don't need automated reviews merging things because it lexically matched sth sensible.

What they need is sustainable maintenance. And if every 10th dollar that went solely into paying for LLMs to write Rust code all day, went instead into the maintainers fund, we'd probably have money left to spare.

14

Short of rejecting most code without review, the only conceivable solution is to use AI to do most of the work of reviewing.

By applying our human creativity we can think of other solutions besides substituting AI automated review for human reviews. The OP seems to have some good ideas.