Stop Searching For the Perfect Model
Sunday, Nov 15, 28 min, about 10 min to read
Theo-brain watched all 28 min
Stop matching models to tasks: Theo keeps one default, Opus 5.5, and has models from other labs review risky changes. He calls that review Council: pick a minimum score for the work, require every reviewer to reach it (9.5 in his example), and stop if the rounds go in circles. Review loops often use up his weekly limit on Muse's $200 plan in two to three days. Skip Council if you pay per-token API prices.
Skim itSkim the Council examples to see how Theo sets the review score and uses several model families to check a change.
Tap any time to jump the video there. Quotes are YouTube's automatic captions, word for word, so some names come out misspelled.
Key takeaways
Don't route models by task. Instead, bring every model's different training and way of exploring into review on risky changes. The variety is the value.
Rather than making sure you use the right model for the right task, I want to make sure all of the models and their unique weights, their unique trainings, their unique ways of exploring things are brought in on hairy changes.
His real Council runs: Opus, the model that wrote the code, first rated it 7.4/10, and it averaged 8.9 after two or more rounds. Of 144 reviewer scores, 128 ended at 9 or higher.
…was started at 7.4 out of 10 and after two or more scored rounds, it would end average 8.9 out of 10.
Across all of the 144 reviewer scores, 21 times it started at below a seven and only twice was it still below a seven at the end. But of all of the work I have done, 128 of the times reviewers came in like this. They ended at a nine or higher.
Unlike code review bots that read a pull request (a proposed code change) after it is filed, Council reviewers work on the same computer. They can run and test the code and ask the main agent questions.
You know how powerful it is to give the agent the computer you wrote the code on to test it there? All three of these agents are able to run the code, to test the code, to ask questions about the code and get an answer from the parent agent. They can do things.
The real goal is that the agent only comes back when the code is ready to merge. Theo says about 80% of the PRs his threads open can now be merged right away.
…roughly now at the point where when I open a PR that one of my threads made, there's an 80-ish percent chance I can merge it right then.
It is expensive. Theo hits his weekly limit on Muse's $200 plan, often within two to three days, just from Council feedback loops.
I have not been using Muse for my day-to-day work a whole lot, but I've been using it inside of the Council a lot. I am hitting my weekly limit on Muse every single week, often in like two to three days on the $200 plan just from doing these feedback loops.
Skills in this one.
Council skill
- What it is
- A skill where the coding model calls reviewers from other model families to score a change from 1 to 10. You pick a minimum score depending on the work, and every reviewer must reach it (9.5 in his example).
- Why Theo cares
- He says he has been blown away by how useful it is to have different model families give feedback on changes in a scoped, reasonable, measurable way.
- How to try it
- Ask your agent to set up a review step that calls models from other model families on a change.
- Have each reviewer score from 1 to 10, and pick a minimum depending on the work (e.g. 9.5) that every reviewer must reach.
- Cap the number of review rounds and tell the agent to stop if rounds go in circles and to keep the scope small.
…score between a one and a 10, where a one is absolute garbage, just throw this away, and a 10 is I cannot fathom why you wouldn't merge it. And depending on what I'm working on, I pick a minimum score.
…need to put a cap at the end here of only go a certain number of runs and or a tap out if you feel like we're not going anywhere and we're running in circles. Do not let repeated reviews result in scope creep. It's your job to fight to keep the scope limited.
Sol loop: one model writes, another reviews
- What it is
- Theo's earlier, simpler version of Council. Opus makes the changes, then GPT-6.1 Sol reviews them. They go back and forth until both agree the change is good.
- Why Theo cares
- He found Sol incredibly thorough at reviews: it would tear every little thing to pieces. While he was building T3 Code's guest-share feature, the Sol loop caught three real security holes.
- How to try it
- Have your main agent write the change, then hand it to a different, more nitpicky model to review.
- Tell them to keep going back and forth until both agree the change is good.
…very minimal version of this where I found that GPT-61 Sol was incredibly thorough at reviews. It would tear every little thing to pieces. So I'd have Opis make changes and call a delegate with Soul to review the changes and give feedback and then they go back and forth until both agree the changes are good.
Soul loop was super helpful when I was building this and it caught three real security holes.
Custom scoring bar for reviewers
- What it is
- Instead of asking reviewers how good a change is, ask them to score something you care about. One example: how likely Theo is to be okay with it. Another: score it several ways at once.
- Why Theo cares
- It is a lossy but powerful way to steer lots of agent work toward your taste. For a T3 Code user, Corey Quinn, he told Council its job was to make sure the PR was in a state Corey would be happy with.
- How to try it
- Write down how you like things built, so reviewers can learn your taste.
- Ask reviewers for separate scores, e.g. how severe the problem is, the risk of future problems, and how likely it is to annoy you.
I had a round where I had to review every PR in three ways. How severe is the problem it fixes? How likely is this to cause problems in the future? And how likely is this to piss Theo off?
And third and most importantly, I want you to bring Council back in, but this time Council's job isn't to make sure the changes are good. Council's job is to make sure the PR is in a state where Corey would be happy.
Advice
Stick with one default model; Theo uses Opus 5.5. He switches mainly to use up tokens he has left on another subscription, not to match models to tasks.
It is very rare that I swap off of my default model once I've picked what I'm using currently. The main reason I switch between models is because I have tokens to burn from a different subscription. Other than that, I'm usually just sticking with at this point as of filming Opus 5.5.
Don't run Council if you pay per token through the API (pay-as-you-go access). It only makes sense if you stack flat-rate subscriptions.
…you should not use this in its current state if you're paying API prices for your tokens. If $10 of Opus costs you 10 bucks, don't do this. If you are stacking subscriptions, however, that's when this gets powerful.
Cap the number of review rounds and let the agent give up if it is going in circles. Tell it plainly that its job is to keep the scope small.
…need to put a cap at the end here of only go a certain number of runs and or a tap out if you feel like we're not going anywhere and we're running in circles. Do not let repeated reviews result in scope creep. It's your job to fight to keep the scope limited.
Explicitly give agents permission to delete code or scrap the PR, because they are oddly scared to delete things.
This is important because agents need the permission to do things that they're scared of and deleting code for whatever reason agents are very scared to do.
Hot takes
Getting several models from the same company to review is like asking the same person at different times of day: the mood changes, the ability doesn't.
His reasoning: He believes Anthropic's current models, Opus, Sonnet and Haiku, were all trained on data from one larger in-house model, so they are very similar.
…the 55 models from Anthropic are distilled from the same base model that is likely Fable 55 or whatever equivalent they have internally. They used that model to generate data that they trained Opus, Sonnet, and Haiku with, which means all of these models…
What I have noticed over time is that getting different models from the same family to review things kind of just feels like asking the same person at different times of the day to review something.
Opus's reasoning traces (the visible 'thinking' text) are garbage, so don't watch them.
His reasoning: He watched it reason through five wrong things, then do exactly the right thing, after he had already sent a correction. He guesses the traces are hidden on purpose or summarized by a dumber model.
The reasoning traces for Opus are garbage. I don't know if they're intentionally obfuscating them to make the model harder to distill against or if they've a dumber model summarizing or what. The reasoning traces for Opus are [bleep] garbage.
OpenAI's models, especially Sol, won't stop on their own when a review loop stalls. They keep fixing tiny nits until the PR is too big to merge.
His reasoning: He found Opus honors a 'stop if we're not making progress' instruction and OpenAI models don't. So he lowered Sol's reasoning effort from extra-high to high, because at full effort it finds too much.
And Soul in particular will just keep [bleep] going trying to squash every single tiny little nit until it's made a god PR that's just way too big to even be considered to merge.
I'm going to knock down from X high to high because if it goes too hard, it will find too much [bleep] and it won't go very far.
How Theo thought about this
He built Council to test one thing: was Muse's feedback as good as Sol's and Opus's? Instead of a single winner, he found all three regularly made valuable new catches the others might have missed.
I built this skill originally because I wanted to validate how good Muse's feedback was compared to Soul and Opus, and I wanted a single skill to have all three come in and review. And what I found was that all three regularly brought up valuable novel catches and found things others might have missed.
He weighs cost against the time saved. A runaway thread spun up 600+ sub-agents and grew a 1,000-line PR to over 20,000 lines. Still, he says even a 5% better chance a PR is mergeable is worth the tokens to him.
…had one thread that had spun up like 600 plus sub agents in order to go through at least 20 rounds of review like this, and it just wasn't making progress. It took what was supposed to be like a 1,000 line of code PR and it bumped it up to like 20,000 plus trying to address every single edge case one of the agents…
Is even if this is a 5% increase in the likelihood that the PRI open is ready to merge, that's worth the token burn for me.
Sign in to mark what you've watched and keep notes.
Have a question about this one? Ask about this video
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.
