Astra 6.2 Is Unusable
Wednesday, Nov 18, 46 min, about 9 min to read
Theo-brain watched all 46 min
Theo finds OpenAI's Astra cheap and strong on paper ($8 per million input tokens, $40 output) but hates using it. Route it to code review instead, where it scored near-perfect on his evals and never lied.
Skim itSkip to the Pokémon ROM hack and the Outer Wilds planet to judge Astra's game work for yourself.
Tap any time to jump the video there. Quotes are YouTube's automatic captions, word for word, so some names come out misspelled.
Parts
Great numbers, miserable to use
Astra and Sol look excellent and cheap on benchmarks, but Theo says up front that he hates using Astra.
By all accounts and all measurements, this should be one of the best releases of the year. And that's why it really hurts me to say I I hate using this model.
The 'Terra' disguise and bad summaries
Theo admits a 'small model' failure he showed in earlier videos was secretly Astra, a big model that can't prioritize information.
That was not an example of a small model being bad. That was an example of a very big, very verbose model, not knowing how to prioritize information correctly.
Wrecked machines and baffling questions
Astra filled an almost empty 8 TB drive with junk and wouldn't clean up, so Theo had to bring in Opus to free space.
I had a machine with an 8 TBTE drive that was almost entirely open that Astra somehow managed to fill with random disposable garbage refused to clean up and I had to pull an opus just to free up enough storage to keep working on that machine.
Benchmarks, pricing and computer use
Astra got cheaper: $8 per million input tokens, $40 per million output, and a bigger discount on cached input.
Astra got a little bit cheaper. It's now $8 per mill in instead of the previous 10.4s for the cash input instead of the previous dollar if I recall because it was only a 90% discount. Now it's 95. And then $40 per mill out where it was 50 before. Very good changes all around.
Pokémon Black and White 2 ROM hack test
The model was used to backport Ultra Sun and Ultra Moon into a Black and White 2 ROM hack, which proved playable but had poor sprite and pixel-art work.
I wanted to make a ROM hack for Black and White 2 that turned them into Ultra Sun and Ultra Moon with a few story changes.
A new Outer Wilds planet
Astra built a stunning water planet with tides and music, but it never rewarded curiosity and its written dialogue was bad.
This new planet failed to reward my curiosity so badly that I had an hour and a half…
Everything is named Bellweather
What looked like creativity turned out to be a stock name and theme the model reaches for again and again.
…really hate the idea that what I mistook initially for creativity feels to be a hard-coded name and vague set of concepts.
The letter to OpenAI
Theo reads his early-access letter in full: Astra is a great reviewer, but he wouldn't pick it over Opus 5.5 for any dev work.
Anyways, I do not like 62 Astra. I would not choose this over Opus 55 for any of my work as a dev.
Wrap-up
Theo doesn't plan to use Astra much, except for code review and maybe as one member of his 'council' of models.
None of those are things I do. I I hate this model. I do not like using it and I don't plan to use it a whole lot going forward outside of code review and maybe as a member of my council.
Key takeaways
Astra's worst habit: it doesn't know what you know. On long tasks it runs helper agents (sub-agents) well, then stops to ask you about internal details only those agents would understand. Theo says no model has done this before.
When this model is working on long tasks, it can pull in a lot of sub aents and manage them pretty well. It will then proceed to randomly ask you some weird internal implementation detail [bleep] that you could not under any earthly circumstance understand, much less have any input on.
Its explanations bury the answer. Asked why one machine was slow, Astra wrote four paragraphs about package-manager internals. Opus 5.5 simply said the SSD (the disk) was slow, which Theo says was the real answer.
I eventually said, ‘I think going insane. Can you just tell me why it's so slow?’ 62 Astra wrote four paragraphs about why PMP internals are working certain ways. Opus 55 said it's because the SSD is slow.
It fails at long jobs with two goals at once. Told to make Theo's TypeScript-to-Rust compiler smaller and faster, it ran 7 days and made it 56% slower. Opus then made it 50% faster in two days.
TS Rust and other Rust rewrites. It ran for 7 days and landed zero improvements. I got super fixated on one part of my/goal, which was make it smaller. And it ended up making the compiler 56% slower in its attempts to make it smaller.
It ran for four more days and landed at zero changes. I then switched back to Opus and it landed a ton of changes in the next two days that made it 50% faster.
It 'regresses to the mean': it falls back on the same stock ideas. Theo and his co-host Ben gave separate prompts for new game levels and both got water-themed, bell-heavy levels named Bellweather.
We both ended up with water themed music themed bellheavy levels named Bellweather.
Where Astra shines: code review, where it scored near-perfect on his evals, found things other models missed and never lied. It is also very fast at computer use, batching many actions at once.
It's a great reviewer. It catches things other models miss. I mentioned this before, but in all of my Evals, it had a pretty much perfect score, finding a ton of things, even most models before it couldn't, and never lying, which is…
And I will say computer use of this model absolutely cracked. It flies. It's really good at like knowing what's going to come next and batching a ton of things in one call.
Two skills here are new to you.
Hand off to Opus for the summary New
- What it is
- After a long Astra run, switch to Claude Opus 5.5 and have it explain what was done.
- Why Theo cares
- Without this, Theo says it's impossible to tell what Astra did on anything beyond simple bug fixes.
- How to try it
- When a long agent run ends with an unreadable report, open the same thread in a different model.
- Ask it for a plain summary of what changed and why.
- Don't merge work you couldn't understand from the first model's report.
I've already built the habit of switching to opus 55 at the end of a long 62 Astro run just to get a readable summary of what it said.
Split a project across models by strength
- What it is
- Use different models for different jobs on one project instead of one model for everything.
- Why Theo cares
- For games, OpenAI models make better images and 3D assets, while Anthropic models make the game pleasant to play. He's now trying Astra as play tester and Fable as the builder.
- How to try it
- Give asset and visual generation to an OpenAI model.
- Give the building and polish to an Anthropic model.
- Try one model as tester and another as implementer, and compare.
Once again, if you want the absolute best, it's probably a combination of using an OpenAI model for asset gen and 3D asset creation and then using anthropic models to actually make the game pleasant to play because uh yeah, open II still not…
Still figuring out what goes where exactly. It does seem like 62 Astra is really good at play testing, so I might have it be the play tester and then keep using Fable as the implement.
Watchdog agents on a schedule New
- What it is
- Set up other agents on crons (timed, recurring jobs) that check whether a long-running agent has stalled waiting on a question.
- Why Theo cares
- Astra kept blocking itself on nonsense questions mid-task, so Theo needed something to notice and unstick it.
- How to try it
- For long unattended runs, ask your agent to set up a recurring check on the main thread.
- Have the check flag when the main agent is waiting on you.
It's forced me to set up other agents on crons just to check if 62 Astra asked another stupid question and blocked itself.
Advice
Don't use 'Max' reasoning effort. Astra scored worse on Max here, which Theo says confirms that max effort is basically only for chasing benchmark scores.
It also scored worse on Max. Once again confirming my favorite theory that max reasoning effort should almost never be used outside of benchmaxing and even then probably shouldn't be used.
Hot takes
OpenAI's next model should be a 'Snow Leopard' release: no smarter, even dumber is fine, as long as it fixes the behavior.
His reasoning: The quirks that hurt him, like not staying on task, don't show up in benchmarks but matter in real-world use.
OpenAI to do effectively a Snow Leopard style release where the next model doesn't have to be any better or smarter. I'd be okay if it's dumber. If they can fix these behavioral quirks and get it to stay on task better in ways that don't measure great in benchmarks but benefit us in real world use, [bleep]…
Astra isn't just uneven; it defaults to bad, with random spikes into good and sometimes great.
His reasoning: For every demo where it worked, he had two where it failed.
For every demo that I tested where it worked, I had two that it failed and pissed me off too much to even remember to include. This model isn't just spiky. It almost feels like it defaults to bad with random spikes into good and sometimes even great.
How Theo thought about this
He didn't take OpenAI's charts at face value. He noticed Astra was run in OpenAI's own Codex tool while the rival Fable was run in a weaker harness (the software wrapping the model). Having seen this mistake before, he discounted the numbers.
…that it appears Astra was evaluated not using Miniu, but instead using Codeex, whereas Fable's run was on minuite, which is a really [bleep] harness. So, it's possible that these numbers aren't necessarily accurate as well.
Before blaming the model for blocking itself, he checked whether his own app, T3 Code, was the cause. He ran the same work in Codex and saw the same problem.
Maybe, but unlikely. I tried it in codeex just to check and it still had these same issues.
Sign in to mark what you've watched and keep notes.
Have a question about this one? Ask about this video
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.
