Self-improving content: applying Karpathy's "autoresearch" loop to marketing

On March 7th, Andrej Karpathy uploaded 630 lines of Python to GitHub and went to sleep. By morning, an AI agent had run 50 experiments on a small language model, found a better learning rate, and committed the results to Git. No human touched anything.

Over the following two days, the agent ran roughly 700 experiments on code Karpathy had already spent months hand-optimising. It found 20 improvements he’d missed. Stacked together, they cut training time by 11%. Fortune called it “The Karpathy Loop.” GitHub gave it 37,000 stars in a week.

But the thing that caught my attention was a single line buried in his announcement: “any metric you care about that is reasonably efficient to evaluate can be autoresearched by an agent swarm. It’s worth thinking about whether your problem falls into this bucket too.”

I’ve been thinking about it since. And I think marketing is one of the best domains this pattern maps onto. Possibly better than the ML use case it was built for.

The pattern is absurdly simple

Three files. That’s it. The entire system runs on three files:

prepare.py is the evaluator. It defines what “better” means. The agent can’t touch this file. It’s locked.

train.py is the thing being improved. The agent edits this file, runs an experiment, and checks whether the change helped. If yes, git commit (keep). If no, git reset (discard).

program.md is the human’s contribution. Written in plain Markdown. It tells the agent what to optimise and what constraints to respect. Karpathy described this as “programming in Markdown” - the human’s job is to write the goal, not run the experiments.

The loop then runs on its own. At around 12 experiments per hour, a single overnight session produces roughly 100 iterations. The agent reads the current state of the file, forms a hypothesis, makes one change, evaluates the result, and either keeps or discards. Then it does it again. And again.

The speed matters, but the compounding matters more. Each round is informed by all previous rounds. The agent isn’t restarting from zero. It’s building on confirmed learnings, which means the hundredth experiment is meaningfully better-directed than the first.

Shopify’s CEO, Tobi Lutke, tried the same pattern the same week. He pointed it at an internal AI model overnight: 37 experiments, 19% performance gain. Then he aimed it at Liquid, Shopify’s templating engine (the code that renders every Shopify storefront), and got 53% faster rendering from 93 automated commits. A SkyPilot team gave it 16 GPUs instead of one and ran 910 experiments in eight hours, watching the agent teach itself to use different hardware for different stages of the search.

These are all ML or engineering applications. But the loop doesn’t care about the domain. It cares about the structure of the problem.

Why marketing fits this pattern better than you’d expect

Marketing optimisation is, at its core, a search problem. Teams search through possible messages, audiences, formats, and offers looking for combinations that work. The space is enormous and the variables are controllable and measurable.

Autoresearch was built for this kind of environment. But marketing has four specific properties that make it an unusually good fit:

Reply rates, click-through rates, conversion rates, cost per acquisition. These are numbers, not feelings. The agent can read them directly.

Feedback arrives fast, too. Email gives you signal in 24-48 hours. Paid ads in 24-72 hours. Landing pages even faster with sufficient traffic. Compare that to scientific research, where you might wait months.

Experiments cost almost nothing. Generating a copy variant is fractions of a penny. Sending an email to a test segment is close to zero.

And the deployment layer is programmable. Meta, Google, email platforms - they all have APIs. The agent can push changes without anyone clicking buttons in a dashboard.

Eric Siu, founder of ad agency Single Grain, was one of the first to do this arithmetic publicly: most marketing teams run 30 to 50 experiments per year. Maybe 52 if they’re ambitious. One per week. The autoresearch pattern enables 36,500+. The individual tests aren’t faster - it’s the human overhead between tests that compresses to almost nothing.

I keep coming back to that number. 30 experiments per year versus 36,500. The gap is structural, not incremental. It changes how the work gets done.

Where it works right now

The canonical starting point is cold email. The metric is clean (reply rate), the modifiable artifact is the email itself (subject line, opener, value proposition, CTA, length), and deployment is fully automatable through tools like Apollo, Instantly, or Lemlist.

The loop runs like this: the agent generates variants, deploys them to matched segments via API, waits 24-48 hours, pulls the reply rate data, keeps winners, reverts losers, and generates the next round. Teams applying this pattern have reportedly seen reply rates move from a 2-4% baseline to 8-12% over four to six weeks of continuous iteration. Those are directional figures, not guarantees. But even half that improvement would be significant for most outreach programmes.

What’s more interesting than any single result is what the agent’s reflection log looks like after a few dozen cycles. It reads like a research finding: “Subject lines framed as a specific problem outperformed questions by 23% in open rate. Emails under 80 words had 15% higher reply rates. ‘Recent news’ personalisation drove strong opens but didn’t convert to replies - it’s an attention tool but not a trust-builder.”

That finding didn’t come from a strategist’s intuition. It came from running the experiment 50 times and reading the data.

The meta-application: optimising the AI tools themselves

This is the one I find most useful for my own work. If you’ve built AI-powered skills, system prompts, or content generation workflows, those prompts themselves can be autoresearched.

The eval criteria become the quality standards for the output, implemented as binary, machine-gradeable assertions. Does the headline include a specific number? Is the copy free of buzzwords? Does the CTA use a specific action verb? Is the total copy between 80 and 150 words?

There’s a documented example from MindStudio that makes the compounding effect concrete: a landing page copy skill started at a 41.3% baseline score against five binary criteria. Round 1 identified “specific number in headline” failing 80% of the time, added a rule, and the score jumped to 68%. Round 2 added a banned buzzword list: 79%. Round 3 added a worked CTA example: 90%. Round 4 tried tightening the word count, score dropped to 82% (regression caught automatically, change reverted). Final result: 41% to 92% in four rounds, three changes kept, one reverted.

That’s the autoresearch loop running on its own tooling. I find something satisfying about the recursion there. You’re using the pattern to improve the pattern.

The AEO angle nobody’s talking about

Here’s where I want to go slightly off-script from the “autoresearch for marketing” conversation that’s been happening on LinkedIn this month.

Answer Engine Optimisation (the practice of structuring content so it gets cited by ChatGPT, Perplexity, Claude, Gemini, and Google’s AI Overviews) has something most marketing channels don’t: something close to an objective eval function.

You can actually query the models and measure whether your content appears in the response. That’s closer to Karpathy’s val_bpb (validation bits per byte) than anything else in marketing. It’s a binary: were you cited or weren’t you?

The mutable file in an AEO autoresearch loop is the content itself - heading hierarchy, answer formatting, entity definitions, claim density, schema markup, source attribution patterns. The agent modifies these elements, publishes the content to a crawlable URL, and then queries the AI engines to check whether citation rates changed.

Tools like LLMrefs track keyword visibility across ChatGPT, AI Overviews, Perplexity, and Claude, automatically generating fan-out prompts based on real user conversations rather than synthetic test queries. That tracking data becomes the eval function. Share of voice across engines for a target keyword cluster is your val_bpb equivalent.

The feedback loop is slower than email (weekly tracking cycles rather than 24-hour reply windows), which means the overnight-experiment cadence doesn’t apply directly. But you can run many keyword clusters in parallel, each with its own content page as the mutable artifact, each evaluated on a weekly cycle. Over a month, that’s 40-80 experiment cycles across a portfolio. Still far more systematic than anyone doing AEO manually.

There’s a separate approach using services like Prompting Company, which create dedicated AI-facing versions of your pages: clean markdown, stripped of UI clutter, with more structured metadata for LLM parsing. The human-facing page stays untouched while the AI-facing version becomes the experiment surface. The agent can iterate aggressively on content structure without risking the user experience.

What makes this interesting is the cross-engine signal. Content that gets cited by Claude, ChatGPT, and Gemini is robust. Content that only works for one model is probably exploiting a quirk. An autoresearch loop naturally converges toward patterns that work across engines, which is the more durable strategy.

I’m genuinely uncertain about how fast this space will move. AI visibility tracking tools barely existed 12 months ago. The eval functions are still maturing. But the fit between autoresearch and AEO is cleaner than most other marketing applications, precisely because the feedback signal is so direct.

The eval function is the whole game

I’ve deliberately saved this for late in the piece because it’s the thing that determines whether any of this actually works or produces confident-sounding noise.

Karpathy has val_bpb: an objective, fast, unambiguous metric. Marketing mostly doesn’t have that. Open rates can be gamed by curiosity-bait subject lines that generate clicks from the wrong audience. CPC optimisation can attract traffic that never converts. A 12% cold email reply rate composed of confused or disinterested replies is worse than a 5% rate from buyers who actually want to talk.

The loop will optimise hard for whatever metric you give it. And if that metric is a poor proxy for the thing you actually care about, the loop will produce beautifully optimised garbage.

Eval criteria must be binary. “Is the copy compelling?” is not something an agent can score. “Does the headline contain a specific number?” is. Yes or no. No sliding scales. Three to six criteria per loop - below three and the agent finds loopholes, above six and it starts gaming the checklist rather than improving the actual output.

Never change them mid-loop. If the baseline comes back at 35% and the instinct is to soften the criteria, resist. Changing the evaluation mid-run invalidates all prior data. If the criteria are wrong, stop, fix them, and restart. And one variable per round. Testing five things at once means you can’t attribute what caused the change.

The real moat is the memory layer

Every individual use case I’ve described is useful on its own. But the thing that compounds into a real competitive advantage is the accumulated findings log across all of them.

A properly designed memory layer records, for every experiment: what was tested, what it was tested against, how it performed, to what confidence level, and why the agent proposed it in the first place. Before generating each new hypothesis, the agent reads recent entries. “We’ve tried three CTAs emphasising speed - none improved conversion. Next, we try urgency-based language.”

Where this gets most interesting is cross-channel learning. If the email loop consistently finds that a specific problem framing outperforms others, that framing becomes the first hypothesis to test in paid ads. If the ad loop finds a strong audience segment, that becomes a targeting signal for email segmentation. Over months of continuous operation, this compounds into something that can’t be easily replicated by competitors who are still running experiments manually.

The loop itself is simple (the code is open source). The moat is the accumulated map of how your specific market responds to your specific messaging across your specific channels.

What stays human

The shift from manual experimentation to autoresearch doesn’t eliminate the human. It moves you to the places where your judgement is irreplaceable.

The agent can test executions of a positioning, but it can’t determine whether the positioning is correct in the first place. What problem does the brand claim to solve, for whom, and why should anyone believe it? That’s yours. Same with offer design, channel selection, and the interpretation calls that matter most - when the loop consistently finds that a segment doesn’t respond, the agent can’t tell you whether that’s a targeting problem, a messaging problem, or a product-market fit problem.

And brand tone. An agent optimising purely for conversion will drift toward whatever pattern most reliably generates clicks. Without explicit constraints in the program.md, that may include tactics that are off-brand or manipulative. The brand guidelines layer must be locked, like prepare.py, and not subject to the optimisation loop itself.

Karpathy himself put it well: the human contribution doesn’t disappear. It becomes “sparse and rare” but qualitatively more important. You write the goal. The agent searches for the answer. You review the morning report and make the decisions the data alone can’t make.

The brief is the product now

The most important investment is writing a good program.md. The tooling matters less than the brief. “Improve our marketing” is useless. “Maximise positive reply rate from VP Engineering at 100-500 person SaaS companies, keeping tone confident but not pushy, testing subject line angle first while holding everything else constant, declaring a winner only when statistical confidence exceeds 90% with a minimum 75 sends per variant” - that’s a brief an agent can actually work from.

Time invested in that document directly multiplies the quality of every experiment the agent runs. The program.md is the human’s real product now. Not the copy. Not the experiment. The brief.

I don’t know yet whether autoresearch-style loops will become standard practice in marketing within a year or three. But the logic is sound: if you have a metric, a modifiable artifact, and a fast feedback loop, the pattern works. And marketing has all three.

The question that matters is whether the brief you write for it is any good.

If you’re exploring how AI automation and self-improving workflows fit into your content or marketing operations, reach out through cowritten.ai.