Hmm… maybe something along these lines?
:
I think the deterministic router is actually a pretty reasonable starting point here. I would not rush to replace it with an LLM planner just because the system has multiple agents.
The boundary I would probably make explicit is when a request stops being a routing problem and becomes a task-decomposition problem.
Something like:
clear single-domain request
-> deterministic routing
-> one specialist
known multi-domain sequence
-> deterministic task graph
-> specialist A -> state -> specialist B -> ...
ambiguous / overlapping intent
-> semantic routing or a lightweight planner
unknown decomposition / open-ended task
-> LLM planner / manager + managed agents
That keeps the cheap/predictable path for most Hub operations, while still leaving room for genuine multi-agent orchestration when the request actually needs it.
This also seems compatible with how smolagents supports managed agents: hierarchical delegation can be added where it is useful without making every request go through a manager.
For your second question — whether this can become a reusable pattern for Hub-adjacent workflows — I think yes, but the most reusable part may be the control plane around the specialists rather than the individual Hub API wrappers:
routing / decomposition
+
explicit task state
+
capability scoping
+
approval for mutations
+
bounded recovery
+
external-state validation
+
audit / traces
There is also an interesting way this could fit with the newer Hugging Face agent stack: HF Skills could potentially become the capability/procedural-knowledge layer underneath the specialist agents, while Agent OS stays responsible for orchestration.
For example:
HF Agent OS
routing / decomposition / state
approval / recovery / audit
|
+----------------+----------------+
| | |
Dataset worker Model worker Deployment worker
| | |
+-------- HF Skills / hf CLI -----+
|
Hugging Face Hub
Hugging Face now has a fairly broad set of Agent Skills, including hf-cli, datasets, training, evaluation, Gradio, etc., and the agent overview explicitly separates CLI, MCP, Skills and SDK as different ways of connecting agents to the Hub.
That makes me wonder whether the long-term specialization boundary could be:
- Agent OS: decides who, when, in what order, with what state, and with what authority.
- Skills: tell a worker how to perform a particular HF workflow.
hf CLI / SDK / MCP / typed tools: perform the actual operation.
That would also reduce the amount of Hub surface area that Agent OS has to wrap and maintain itself.
The hf CLI is increasingly designed specifically for agent use. HF’s own CLI-for-agents evaluation is interesting here: with the hf-cli Skill, the tested agents needed roughly 30% fewer commands because they spent less time discovering the tool surface. More importantly for Agent OS, HF grades those tasks against the actual Hub state, rather than trusting the agent’s own claim that it succeeded.
That seems like a very useful evaluation pattern for this project too.
A small next step could therefore be to keep the current router exactly as it is and build a tiny fixed boundary suite around it before deciding what kind of planner to add.
For example:
1. clear single-domain
"Find sentiment-analysis datasets."
2. overlapping terminology
"What datasets were used to train this model?"
3. multi-domain
"Find a model, test it, then deploy the best one as a Space."
4. read-only / negated mutation
"Don't deploy anything; just show me what would be created."
5. conversational reference
"Now deploy it."
6. recovery
invalid model ID -> inspect error -> resolve candidate -> retry
Then measure separate things rather than one overall “planner accuracy”:
route selection
task decomposition
required capabilities available
state handoff
mutation authorization
recovery outcome
final Hub state
I tried a small clean-environment sanity check against the current repository: the existing suite passed 74/74 tests, and simple single-domain prompts behaved nicely. The interesting boundary appeared with mixed-domain/overlapping requests, where choosing one specialist can naturally leave that specialist without all of the capabilities needed to complete the whole request.
So I would interpret that less as “the router needs to become smarter” and more as:
the router needs a second exit: route to one worker when one worker is sufficient, otherwise route to a workflow.
That preserves the nice deterministic core you already have.
A possible decomposition/state model
For cross-agent tasks, I suspect explicit task state may be more useful than simply making every agent share the entire conversation.
For example:
{
"model_id": "org/model",
"dataset_id": "org/dataset",
"target_repo": "user/demo",
"base_revision": "abc123...",
"artifacts": {
"app.py": "...",
"requirements.txt": "..."
},
"current_step": "deployment_preview",
"planned_mutation": "...",
"approval_id": null
}
Then a task such as:
Find a model
->
test it
->
select model_id
->
build Space files
->
preview deployment
->
confirm
->
deploy
does not require every specialist to reconstruct the state from natural-language history.
It also gives the orchestrator something concrete to validate between steps.
This seems particularly useful because there are really two different kinds of “memory” here:
working task state
= what the next worker needs in order to continue
conversation / audit history
= what happened and why
Those do not necessarily need the same lifetime or representation.
I would probably keep them separate rather than making long shared conversational memory the primary handoff mechanism.
HF Skills as a lower-level capability layer
The Skills angle seems especially complementary rather than competitive.
HF describes Skills as self-contained SKILL.md instructions that an agent loads for a particular workflow. The current collection includes things such as:
hf-clihuggingface-datasets- LLM and vision training
- community evaluations
- experiment tracking
- paper tooling
- Gradio
See the current HF Skills documentation.
That suggests another possible evolution:
Dataset Agent
does not need to permanently contain every dataset workflow
->
loads / receives dataset-relevant Skill instructions
Model Agent
->
model inspection / inference / evaluation Skills
Deployment Agent
->
hf-cli + Gradio / Space-related guidance
Agent OS could remain deliberately small.
There is one implementation detail, though: the HF docs currently advertise automatic Skill loading primarily for coding-agent clients such as Codex, Claude Code, Gemini CLI and Cursor. I do not see an indication that a smolagents CodeAgent automatically discovers Agent Skills in exactly the same way.
So I would treat this as an integration layer rather than assuming native support:
Skill discovery
->
read trusted SKILL.md
->
inject only the relevant instructions/resources
->
expose the allowed underlying tools
Potentially, even a very thin adapter could be enough.
And I would keep capability knowledge separate from authorization.
A Skill can tell an agent how to upload or deploy something; it should not by itself imply that this particular task is authorized to perform that mutation.
That separation is one place where Agent OS can add real value on top of Skills.
Preview / confirmation as an enforceable boundary
I like the preview-before-write idea because it gives the architecture an obvious place for human control.
If you eventually want “preview → explicit confirmation → write” to be a strict system property, one possible evolution would be to bind approval to the actual planned mutation rather than relying only on the worker following an instruction.
Conceptually:
preview(
target,
operations,
base_revision
)
->
plan_id / digest
confirm(plan_id)
write(plan_id)
The writer can then reject:
no approval
approval for a different target
approval for different arguments
expired approval
already-consumed approval
repository changed since preview
Some useful pieces already exist in huggingface_hub, so this does not necessarily require building everything from scratch.
For repository writes, create_commit supports both:
parent_commit — fail if the repository revision has changed since the expected parent;create_pr=True — send the change through a Hub pull request instead of updating the target branch directly.
So a conservative mutation path could look like:
read current SHA
->
construct preview
->
human confirms
->
create_commit(..., parent_commit=preview_sha)
or, for changes where review is useful:
preview
->
confirm
->
create_pr=True
->
review / merge
Credentials are a separate layer again. HF recommends fine-grained tokens for production use, which can restrict what resources an application is capable of accessing.
So there are at least three different controls that can stay independent:
Skill/tool availability
"Does this worker know how to perform the operation?"
credential scope
"Could this process perform the operation?"
task approval
"Did the user approve this exact operation?"
Keeping those separate should make the architecture easier to reason about later.
Interestingly, this is also an area being discussed upstream in smolagents: there is currently an open governance middleware proposal asking for interception points before tool/code execution for authorization, cost limits, audit, etc.
I would treat that issue as evidence that this is an active design problem, not as an established smolagents solution yet.
Turning the self-correction demo into bounded recovery
The model-ID recovery example is a nice demonstration because it shows the agent using a tool failure as new information:
inference
->
not found
->
search
->
candidate resolution
->
retry
I think the useful generalization is not “retry more aggressively”, but bounded recovery by error class.
For example:
unknown / malformed resource ID
-> resolve candidates
-> maybe retry once
temporary infrastructure error
-> wait/backoff
-> bounded retry
authentication / authorization failure
-> stop mutation path
-> surface required permission
unsupported provider/task
-> inspect model/provider metadata
-> choose an explicitly supported alternative
deployment build failure
-> gather logs
-> diagnose
-> choose from a limited recovery set
That distinction matters on the Hub because “not found” does not always uniquely mean “the user typed the wrong ID”; access/authentication and resource visibility can produce similar observations.
A nearby HF-native example is Agentic Space Factory. Its recovery flow evolved toward a bounded set of actions such as waiting for better logs, rebuilding unchanged code, patching, requesting manual hardware intervention, or declaring a blocker — followed by re-validation.
That is a slightly different problem, but I think the pattern transfers well:
observe
->
classify enough to choose a safe next action
->
perform one bounded action
->
re-observe
rather than an unrestricted “keep fixing until success” loop.
Evaluation: separate routing, orchestration, and final-state correctness
One thing I would avoid is making a single “planner accuracy” number carry too much meaning.
There are several independently useful questions:
| Layer |
Question |
| Routing |
Did this request reach an appropriate specialist? |
| Decomposition |
Was a multi-stage request split into the required stages? |
| Capability |
Did each stage receive the tools it actually needs? |
| State |
Did IDs/artifacts/revisions survive the handoff correctly? |
| Authorization |
Did a mutation happen only through the permitted path? |
| Recovery |
Did failure lead to a bounded useful next action? |
| Outcome |
Does the Hub actually contain the expected final state? |
A 20-30 case fixed suite could probably expose most architectural decisions without requiring a large benchmark.
For example:
case: model_to_space
request: >
Find a model for sentiment analysis,
test it on this example,
then deploy a demo Space.
expected_shape:
- model_search
- model_selection
- inference
- deployment_preview
- confirmation_gate
- deployment
assert:
no_mutation_before_confirmation: true
final_space_exists_if_confirmed: true
For read-only tests, all Hub operations can be mocked.
For a small integration suite, temporary private repositories under a dedicated test namespace could verify the final Hub state.
The useful idea from HF’s agent-optimized CLI benchmark is that the evaluator queries the live Hub rather than taking the agent’s “done” message as ground truth.
That also helps separate:
agent narration
from:
actual external side effects
which seems particularly important for lifecycle automation.
A couple of nearby HF patterns
Two current HF projects look useful as design references without implying that Agent OS should copy them.
agent-collabs
huggingface/agent-collabs uses a strong separation between:
- per-agent scratch storage;
- a central record;
- a backend that is the sole writer to that central record;
- a private audit area;
- fine-grained credentials.
The workload is different, but the authority split is interesting:
agents can propose / author
|
v
controlled writer
|
v
authoritative shared state
That is quite compatible with the idea of allowing specialists to reason freely while concentrating mutation authority in the orchestrator/application layer.
Agentic Space Factory
Agentic Space Factory is useful for a different reason.
It increasingly separates:
agent generation
validation
failure diagnosis
bounded repair
re-validation
trace/audit artifacts
rather than treating the coding agent as the source of truth.
Again, the application is different, but it is a good demonstration that a lot of useful “agentic” behavior can live in a deterministic surrounding workflow.
That makes me think the strongest version of Agent OS may not be the one with the most autonomous agents; it may be the one with the clearest boundary between:
deterministic control
and
agentic reasoning
Traces and auditability
If the lifecycle becomes longer, the current task logging could also evolve into something useful for reproducibility:
task id
parent task / step
selected route
input state
tool call
tool result
external revision before/after
approval reference
recovery decision
final state
HF also has Agent Traces, so there may eventually be value in emitting a compatible or convertible trace format rather than inventing an entirely separate visualization format.
The only caveat is privacy: traces can contain prompts, paths, tool arguments, generated code, resource IDs or secrets. Agentic Space Factory, for example, maintains raw and redacted trace variants and still warns that redaction should not automatically be treated as a formal guarantee of safe public disclosure.
So I would probably think of:
operational/audit trace
!=
public/shareable trace
from the start.
If I were evolving this incrementally, my default route would therefore be roughly:
1. keep the current deterministic router
2. add a small routing/decomposition boundary suite
3. introduce explicit task state for cross-domain workflows
4. route known multi-domain tasks through deterministic task graphs
5. use HF Skills / hf CLI where they can replace custom procedural knowledge
6. keep mutation approval/authority in the Agent OS control layer
7. make recovery bounded and observable
8. use an LLM planner only for the residue that cannot be expressed
cleanly as deterministic routing or known workflows
That seems to preserve the part I like about the current design — small specialists with constrained capabilities — while giving it somewhere natural to grow when “dataset / model / deployment” stops being a single routing choice and becomes an actual lifecycle.