Rendered at 17:05:22 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
theptip 34 minutes ago [-]
Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that.
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
tyleo 5 hours ago [-]
The chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.
layer8 2 hours ago [-]
It isn’t inverted though? 32000 is at the bottom, 40000 is at the top. Maybe it changed since you commented.
couAUIA 5 hours ago [-]
well thank you so much for this
sreekanth850 45 minutes ago [-]
Anthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and layered architecture). After moving to codex, life has been much easy and free form usage anxiety. Now managing entire things with 2 codex plus account per team member. Its high time anthropic should stop scare mongering and build efficient models. Everyone doesn't need Fable. People need models that solve problems efficiently.
gmac 36 minutes ago [-]
I've found Opus 4.8 pretty amazing, and Codex a bit meh. YMMV?
sreekanth850 33 minutes ago [-]
There is one thing that stay apart, I can use luna high with pretty less cost and do a comprehensive audit using sol high on every push from github.
Tenoke 4 hours ago [-]
Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
esperent 3 hours ago [-]
This is one of the advantages of pi. I made a /protect command that protects the message from compaction. I also protect skills automatically.
So for long running tasks I'll do
/protect your goal is...
jswny 4 hours ago [-]
Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction.
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
brookst 3 hours ago [-]
Compacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction.
Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.
It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
tony_cannistra 1 hours ago [-]
This is the way.
mrtesthah 2 hours ago [-]
Most models’ reasoning abilities drops off significantly between the 256K-1M token ranges of the context window. There’s too much stuff to “pay attention to” at once.
hedgehog 31 minutes ago [-]
I auto compact around 200k tokens both due to this and because the cached read cost really escalates when sessions have more tokens than that (too short and you pay a lot in per-compact re-reading of state)
weiliddat 3 hours ago [-]
One interesting harness thing I saw Cursor do is to give the model access to the entire thread.
Even if it doesn't fit in the context window, the model can search through past turns and sanity check if something doesn't seem to be going right, or be prompted to follow an early message, "when starting on a new item, review the first message for how we should approach this"
semiquaver 4 hours ago [-]
Unlike other model/harness pairs, codex+gpt also passes an opaque encrypted artifact speculated to be an embedding representing the conversation back to the successor generation which is “denser” or at least higher fidelity than summarized text.
jedisct1 2 hours ago [-]
This is not an issue with Claude, but with Claude Code.
With swival.dev you can use long sessions without ever doing any manual compaction or reset, even with Claude models.
3 hours ago [-]
Androider 3 hours ago [-]
Quality degrades long before you hit compaction, compaction is the "Next gas station, 100 miles" sign but you're already in the middle of nowhere.
You don't need to go down the rabbit hole of crazy workflows, but to avoid slop:
- Break down the work into tasks
- New context. Create a plan for one task.
- New context. Implement the plan.
- New context. /code-review the implementation
- New context. Fix the review findings.
- Repeat for next task.
I do this with Fable 5, and the quality is consistently quite good. If the context goes over 50%, the quality will become crap and you end up with 4 duplicates of the same thing across the codebase. Letting a current session review its own work is like asking a student to grade their own paper.
beefsack 3 hours ago [-]
For the vast majority of models I've worked with, 150k tokens of context seems like some magic spot. If you can stay under it then the session will stay on track and the agent won't start getting forgetful and weird.
Just like with real work, it's valuable to break down big tasks into small tasks that you can knock over in a single session. When a session does start getting too big, you just need to ask the agent to reply with a comprehensive handover report and paste it into a new session.
robwwilliams 1 hours ago [-]
For my quirky workflows I need 100K even to prep Opus 4.8 or higher to begin good work. Not programming, but a complex synthetic architecture for implementing more attentional autonomy for machines.
IceDane 4 hours ago [-]
Stop using compaction. Stop treating sessions as something you keep alive for weeks. Set up a proper workflow for handing off work and start handing off well before maximum context is hit. It's less efficient, slower and more costly to constantly be sitting at high context and compaction is just not good.
robwwilliams 1 hours ago [-]
There is no one-size-fits-all, even for programming.
IceDane 34 minutes ago [-]
There's a good way and a bad way to use tools. My suggestions were based on my extensive usage of the tools in question.
tantalor 4 hours ago [-]
What is /goal?
lambda 11 minutes ago [-]
The article describes it.
Both Codex and Claude Code have it, but they work slightly differently.
Claude Code uses Haiku to read through the transcript and decide if the goal has been completed. If not, Haiku injects a prompt back to the main model to indicate what still needs to be done.
In Codex, instead it's a tool available to the main model, plus some part of the surrounding harness that will re-prompt it if the tool calls haven't yet indicated that the goal is complete.
The issue that they are trying to solve is that sometimes models will stop before they have actually fully completed whatever task they were given; attention isn't perfect, and someitmes they'll complete part of it but not the whole task. Rather than making the user come back and re-prompt to keep going, they add a way to automatically do a bit more nudging to try to get the model to finish the task.
swader999 4 hours ago [-]
On Claude if you start with that, it won't stop until it achieves or exhausts your prompt. It feels like "here's your mission, go do it". I use it a few times a week.
9dev 1 hours ago [-]
I'm way too afraid of a "paperclip problem"-style target overfitting to use it really; how could I even describe a goal so well it doesn't sacrifice other things that are important to me? Like, if my goal is to make an endpoint faster, will it create an over-engineered mess out of the clearly readable code I have to reach that goal?
brap 3 hours ago [-]
> won't stop until it achieves or exhausts your prompt
This is overselling it. In all implementations of this that I’ve seen, a more correct phrasing is “won’t stop until it pinky-promises it achieved your goal”.
It usually relies on something like a hook that refuses to let the agent stop unless it includes a “promise sentinel”, which is basically the model having to include “I swear I’m done” in the response (usually as a Markdown comment so the user doesn’t see it). If that sentinel isn’t in the response, the hook rejects the completion and hands it back to the agent (sometimes it also restates the goal to keep it focused). It’s useful for sure, but it fails many times for obvious reasons.
More robust implementations use the hook to spawn another agent that verifies the goal is achieved, but that too isn’t prefect and sometimes performs worse.
Of course if you have a problem that can be deterministically verified, you could have the hook run this verification instead, but this is usually not the case (and even when it is, agents will often cheat the verification e.g. by deleting a test file).
hungryhobbit 1 hours ago [-]
The key thing is that normally Claude grades Claude's work, so you do get a"pinky promise, as all of Claude's training bias tells it to plunge ahead (even if that means "fixing" a test by commenting it out).
With /goal, a different Claude instance evaluates the goal. This is the robust version you described, and such an "adversarial Claude's" approach, with or without/goal, is exactly how you avoid pinky promises (from one Claude).
jascha_eng 3 hours ago [-]
Is this useful? I feel like the problem is usually not that the model isn't capable of achieving what I give it, but the way it does it. Especially if originally I didn't 100% know how I would do it myself the model often takes weird paths through the code base, takes shortcuts that end up in weird feature interactions or pulls in a dependency without weighting if it could've been done without that.
I haven't really found a good way to solve this other than:
1. Produce an initial PR fulfilling all the requirements I knew at the start
2. Chat with the model about any weird snippets I notice and talk through alternatives
3. Simplify anything that I think is overengineered or plain unncessary
Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.
In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
swader999 3 hours ago [-]
It's useful for things where it just needs to get through to completion. Long running tasks. I walk away and expect it to be done without pausing for input.
jascha_eng 3 hours ago [-]
Can you give an example? And more curious about what you do with the resulting code afterwards I imagine its gonna be a big chunk then?
hungryhobbit 1 hours ago [-]
I feel like "do X untill tests are green" is sort of the prototypical /goal case.
madeofpalk 2 hours ago [-]
Adding a new lint rule that a lot of code violates.
jascha_eng 31 minutes ago [-]
And that doesn't work with a simple prompt?
ryan_n 4 hours ago [-]
The llm runs in a loop until it meets a condition (the goal).
aetch 57 minutes ago [-]
It loops until it thinks it has finished the goal
deepdarkforest 2 hours ago [-]
An agent is an llm running on a loop until it decides it should stop.
/goal is a gimmick where you run a "parent" agent on top that runs the agent on a loop until the it decides to stop, just prompting it "nope, not done yet, continue".
ai_fry_ur_brain 4 hours ago [-]
[dead]
o10449366 5 hours ago [-]
/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
SvenL 2 minutes ago [-]
Isn’t it weird to think a LLM “fully understands” something? What does this mean?
It reminds me of this printing trick “until you are 95% confident”…
Would really be interested to know how this influences the work of the LLM. Also, how would it be if I write “until you certainly understand it”?
embedding-shape 5 hours ago [-]
> 3. /goal Spend X minutes from $time writing a technical design doc on $feature.
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
lonelyasacloud 3 hours ago [-]
Leaving to run unhindered makes sense when the agents understand both the complete domain context and can be supplied with a strong exit criteria that they understand well.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
illliillll 4 hours ago [-]
Well, I’ve been having 5.6 sol work on tasks like “find every OTA app on the internet”
I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.
o10449366 2 hours ago [-]
I think like the other commentator noted, I don't do this blindly, as I also wouldn't if I was instructing a junior engineer. I have a rough sense of how long it would take me to research all angles of the task. 5 minutes for a complicated feature is too short for a comprehensive plan. Letting it run for a few hours is also probably unproductive. But 10-60 minutes depending on complexity with followup review is a nice sweet spot. Just my $0.02.
bob1029 4 hours ago [-]
The hypothesis generation phase is looking like the most critical part of having an agent reliably hit your targets.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
ai_fry_ur_brain 4 hours ago [-]
[dead]
aszen 3 hours ago [-]
Won't surprise me if the llm just calls sleep after it's convinced it knows all
andai 4 hours ago [-]
Results seem mostly noise to me. One eval per model, in a large problem space (i.e. a problem which requires many attempts to solve well).
couAUIA 4 hours ago [-]
Yes I agree, but I actually did a lot more runs, with different prompts, different times ect... And each time /goal had a small or insignificant impact
codelion 2 hours ago [-]
Just use OpenEvolve for such problems.
akoboldfrying 3 hours ago [-]
If you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting).
Both are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.
couAUIA 6 hours ago [-]
A deepdive on the /goal effect on a problem literally made for this.
stevefan1999 2 hours ago [-]
...is this not a Travelling Salesman Problem?
hyperpape 2 hours ago [-]
Related, but not quite the same. The solution to this problem can have more than one loop, whereas TSP definitionally has only one. Not quite sure how that changes the difficulty or the tools that you’d use.
stevefan1999 28 minutes ago [-]
Because my instinct immediately went into simulated annealing or ant colony.
Or I think my PTSD on advanced algorithm course kicked in
Very interesting, will surely help my future projects
techpression 4 hours ago [-]
I love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other.
Then again, maybe CSS is the ultimate benchmark.
thickclearglass 2 hours ago [-]
Codex has a very bad system prompt which no one at OpenAI is seemingly aware of.
Sol frontend is perfectly serviceable if you use it from something like Pi.
cwmoore 4 hours ago [-]
I do not know the whole picture, but if you are asking for blind one-shot CSS, you might benefit from wiring the model to take screenshots of various end-browsers and discuss them as you iterate.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.
techpression 18 minutes ago [-]
I actually have very detailed specifications around the CSS, the problem is that LLM's are trained mostly on codebases with high amounts of complexity so they like to create layer of abstractions that only make your codebase much harder to reason about, machine and man alike.
This is very visible in CSS, where almost every single thing automatically becomes a component wrapping the CSS (be it React or LiveView in my case), which is very often meaningless because the CSS itself is the component.
baq 4 hours ago [-]
CSS is the reason I refuse to do any frontend work except FE infra and I know I’m not alone here, soooo yes I guess?
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
So for long running tasks I'll do
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.
It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
Even if it doesn't fit in the context window, the model can search through past turns and sanity check if something doesn't seem to be going right, or be prompted to follow an early message, "when starting on a new item, review the first message for how we should approach this"
With swival.dev you can use long sessions without ever doing any manual compaction or reset, even with Claude models.
You don't need to go down the rabbit hole of crazy workflows, but to avoid slop:
- Break down the work into tasks
- New context. Create a plan for one task.
- New context. Implement the plan.
- New context. /code-review the implementation
- New context. Fix the review findings.
- Repeat for next task.
I do this with Fable 5, and the quality is consistently quite good. If the context goes over 50%, the quality will become crap and you end up with 4 duplicates of the same thing across the codebase. Letting a current session review its own work is like asking a student to grade their own paper.
Just like with real work, it's valuable to break down big tasks into small tasks that you can knock over in a single session. When a session does start getting too big, you just need to ask the agent to reply with a comprehensive handover report and paste it into a new session.
Both Codex and Claude Code have it, but they work slightly differently.
Claude Code uses Haiku to read through the transcript and decide if the goal has been completed. If not, Haiku injects a prompt back to the main model to indicate what still needs to be done.
In Codex, instead it's a tool available to the main model, plus some part of the surrounding harness that will re-prompt it if the tool calls haven't yet indicated that the goal is complete.
The issue that they are trying to solve is that sometimes models will stop before they have actually fully completed whatever task they were given; attention isn't perfect, and someitmes they'll complete part of it but not the whole task. Rather than making the user come back and re-prompt to keep going, they add a way to automatically do a bit more nudging to try to get the model to finish the task.
This is overselling it. In all implementations of this that I’ve seen, a more correct phrasing is “won’t stop until it pinky-promises it achieved your goal”.
It usually relies on something like a hook that refuses to let the agent stop unless it includes a “promise sentinel”, which is basically the model having to include “I swear I’m done” in the response (usually as a Markdown comment so the user doesn’t see it). If that sentinel isn’t in the response, the hook rejects the completion and hands it back to the agent (sometimes it also restates the goal to keep it focused). It’s useful for sure, but it fails many times for obvious reasons.
More robust implementations use the hook to spawn another agent that verifies the goal is achieved, but that too isn’t prefect and sometimes performs worse.
Of course if you have a problem that can be deterministically verified, you could have the hook run this verification instead, but this is usually not the case (and even when it is, agents will often cheat the verification e.g. by deleting a test file).
With /goal, a different Claude instance evaluates the goal. This is the robust version you described, and such an "adversarial Claude's" approach, with or without/goal, is exactly how you avoid pinky promises (from one Claude).
I haven't really found a good way to solve this other than:
1. Produce an initial PR fulfilling all the requirements I knew at the start
2. Chat with the model about any weird snippets I notice and talk through alternatives
3. Simplify anything that I think is overengineered or plain unncessary
Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.
In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
/goal is a gimmick where you run a "parent" agent on top that runs the agent on a loop until the it decides to stop, just prompting it "nope, not done yet, continue".
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
It reminds me of this printing trick “until you are 95% confident”…
Would really be interested to know how this influences the work of the LLM. Also, how would it be if I write “until you certainly understand it”?
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
[0] https://neos-server.org/neos/
Or I think my PTSD on advanced algorithm course kicked in
Edit: looks like the closet reduction should be https://en.wikipedia.org/wiki/Ring_star_problem with bounded circuit length
Sol frontend is perfectly serviceable if you use it from something like Pi.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.