Refreshing Stale Code Intelligence

QCon London 2026

Session AI/ML

Refreshing Stale Code Intelligence

Tuesday Mar 17 / 01:35PM GMT, Churchill (Ground Fl.) at The QEII Centre, London

Abstract

Coding models are helping software developers move faster than ever, but weirdly, the models themselves are not keeping up. They are trained on months-old snapshots of open source code. They have never seen your internal codebase, let alone the code you wrote yesterday. And as AI-assisted contributions have surged, the data shows acceptance rates for pull requests have actually declined. More code is being written, but more of it is being rejected.

Why? Because every repository has its own unwritten rules, architectural constraints that live in the heads of your senior engineers and in the patterns of your commit history, but that no generic model has ever learned. We set out to measure this gap with data, and what we found is that it is real, growing, and structural.

This talk presents original research on what makes codebases architecturally unique, and gives you a practical framework for closing the gap yourself. You will learn how to surface your own repo's unwritten rules (your senior engineers already know them, they just haven't been written down), how to put those rules where your AI tools can actually use them, and how to evaluate whether your current tooling respects the constraints that matter most. No special tooling required. You will leave with a practice you can start next week with a whiteboard and your most experienced engineers.

Interview

This session is about the growing gap between what AI coding models know and what your codebase actually requires. Every software organization has unwritten architectural rules. This knowledge lives in the heads of senior engineers and in the patterns of your commit history, but no generic coding model has been trained on it. I present original research showing that these rules exist, they are measurably distinct from generic software engineering best practices, and current approaches like RAG and standard linting miss a significant fraction of violations. 

For senior developers and architects, this matters because you are the ones who currently hold this knowledge, and you are the ones feeling the pain when AI-generated code violates it. The good news is that you already have what you need to close the gap. Your senior engineers know these rules; they just have not been written down. This talk gives you the data to understand the problem clearly and a practical, low-tech framework for surfacing those rules and putting them to work in whatever AI tooling you already use.

Two trends are colliding. First, AI-assisted code generation has exploded. Our data shows AI tool mentions in PRs grew 50x between 2022 and 2025 across a number of major open source projects. Second, acceptance rates for PRs are declining, not rising. They dropped from 74.7% to 66.2% over the same period. More code is being generated, but more of it is being rejected. This is not a temporary growing pain. As more of your team's output is AI-assisted, the cost of that AI not understanding your specific architectural constraints compounds. The teams that close this gap fastest will do it by getting explicit about what their codebase uniquely requires: documenting those rules, feeding them into their existing AI tools' system prompts and review checklists, and evaluating their tools against the constraints that actually matter rather than generic benchmarks. This does not require new tooling or infrastructure. It requires a whiteboard session with your most experienced engineers and the discipline to write down what they already know.

The biggest challenge is the gap between generic and specific. Today's coding models know how to write Python, but they do not know that your monorepo requires version synchronization across package boundaries, or that your framework mandates lazy imports for optional dependencies, or that new model implementations must register in a specific registry. RAG helps somewhat: you can put your style guide into the context window. But our research shows that retrieval-based approaches still miss a significant fraction of the violations that repo-specific constraints catch. The deeper challenge is that these rules are usually undocumented. They live as tribal knowledge in the heads of your most experienced engineers. When we extracted constraints computationally from various open source repos, we found real, important rules that nobody had written down. But here is the thing: senior engineers on those projects would have recognized every one of those rules immediately. The knowledge exists. It is just locked in people's heads instead of written where tools and new team members can use it. That is a problem any team can fix without special tooling.

Sit down with your two or three most senior engineers and explicitly catalog your repository's architectural constraints. Ask three questions. First: what are the rules in our codebase that are not written down anywhere? Second: look at recent PRs that got sent back in review, not for CI failures but for convention violations. What was the convention? Write it down. Third: what do you spend time explaining to every new team member? Those are constraints too.

In our research, the results consistently included rules that were real and important but that nobody had documented. Things like "configuration validation must enforce that 4-bit loading and flash attention are disabled during adapter merging" or "the frontend package version and the Python package version must be kept identical." Once you have even a partial list, put the top constraints in your AI tools' system prompts and start reviewing AI-generated code against them. That single practice will close more of the gap than switching to a better generic model.

QCon is one of the rare conferences where the audience is primarily architects and senior engineers who build real systems. That matters for this topic specifically because the people who understand architectural constraints are the people in the room. The value of knowing your repo's unwritten rules only resonates with people who have been burned by violations of those rules. QCon's "no vendor pitches" policy also means I can present research findings honestly, including where our experiments failed in interesting ways, without having to pretend everything is perfect.

Topics

AI/ML agents Code Generators fine tuning
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon London 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Tuesday 17 March

10:35 Whittle (3rd Fl.) Session AI/LLM Reliable Retrieval for Production AI Systems Lan Chu AI Tech Lead and Senior Data Scientist 11:45 Fleming (3rd Fl.) Session AI Rewriting All of Spotify's Code Base, All the Time Jo Kelly-Fenton, Aleksandar Mitic 13:35 Churchill (Ground Fl.) Session AI/ML Refreshing Stale Code Intelligence Jeff Smith CEO & Co-Founder @Neoteny AI, AI Engineer, Researcher, Author, Ex-Meta/FAIR 14:45 Churchill (Ground Fl.) Session AI Beyond Context Windows: Building Cognitive Memory for AI Agents Karthik Ramgopal Distinguished Engineer & Tech Lead of the Product Engineering Team @LinkedIn, 15+ Years of Experience in Full-Stack Software Development 15:55 Fleming (3rd Fl.) Session applied ai Building an AI Gateway Without Frameworks: One Platform, Many Agents Amit Navindgi, Jatin Aneja 17:05 Whittle (3rd Fl.) Session Async Agents in Production: Failure Modes and Fixes Meryem Arik Co-Founder and CEO @Doubleword (Previously TitanML), Recognized as a Technology Leader in Forbes 30 Under 30, Recovering Physicist