For now, I think I may have found one candidate workflow shape:
The shape I have in mind is roughly:
generate / fan out N candidates
↓
gather them
↓
evaluate / select one
↓
send only that result into an expensive downstream step
For example, very close to the use case already in the README:
4 Flux stills → choose one → LTX video
I do not mean that this use case is currently impossible. The existing agent path already handles the semantic version nicely: the agent can inspect the generated images, decide that “the third frame is the one”, keep_output it, and use that asset in the next workflow. That is probably the right boundary when “best” is a genuinely visual or subjective judgment.
The part that seems potentially interesting as a workflow shape is the deterministic version of the same pattern.
My default split would probably be:
- semantic / subjective selection → keep it at the agent boundary, as today;
- deterministic N→1 selection → possibly a small selector/reducer task inside the workflow;
- actual conditional control flow → only introduce that if there are use cases that cannot be expressed cleanly by the first two.
That seems smaller than adding a general when/branch language immediately, and it fits the existing for_each → gather: → task/pipeline structure pretty naturally.
For example, the deterministic case could be something like:
generate 4 candidates
→ gather
→ select argmax(score)
→ run the expensive video/upscale stage only for that candidate
The selector could be as simple as “highest score”, “first above threshold”, “lowest artifact score”, or “use the index returned by another task”. A similar N→1 shape shows up in ComfyUI’s official ImageSelector custom-node walkthrough, so it seems like a reasonably real workflow pattern rather than a hypothetical one.
If selection remains semantic, though, I would not try to force it into the workflow DSL just for completeness. The current agent-mediated two-job flow may actually be the cleaner abstraction.
Separately, I found one small preflight edge case that seems especially relevant when an agent is authoring the JSON.
On commit 80caec73..., I tried the project’s own workflow validator against a few tiny cases. The result was roughly:
| Case |
Validation |
| normal task step |
valid |
unknown step property when |
valid |
unknown step property retry |
valid |
unknown step property select |
valid |
typo relase_pipeline |
valid |
known release_pipeline with the wrong type |
invalid |
missing required step name |
invalid |
unknown pipeline.configuration property |
invalid |
So this does not look like “validation is generally permissive”. It looks more specifically like step-level unknown properties are currently open, while several other layers are checked much more strictly.
That creates an interesting agent-facing failure mode: an agent could invent a plausible-looking property such as when or mistype a real property, get a successful preflight, and then assume semantics that the runtime does not actually implement.
Given that one of the nice properties of this project is catching agent mistakes before model loading / GPU work, I wonder whether step-level unknown properties might deserve at least a warning.
I would not automatically make them hard errors, though. There seem to be a few reasonable policies depending on what you want the workflow format to become:
unknown step fields are intentional extension points
→ keep them open, maybe warn
forward compatibility matters, but typos should be visible
→ warning by default + optional strict mode
the step vocabulary is intended to be closed
→ schema error is probably simplest
JSON Schema itself permits additional properties by default, so the current behavior is not surprising by itself. The question is more whether that default is desirable for an agent-authored DSL.
There is a fairly close failure shape in ComfyUI issue #15669: programmatically supplied unknown inputs can be silently ignored while validation succeeds and the generation still runs. That is a different codebase and not necessarily the same root cause, but I thought it was a useful example of why silent unknown fields become more noticeable once software rather than a human is constructing the workflow.
Why I think selector/reducer is a useful boundary
The current workflow machinery already seems to have most of the surrounding pieces:
for_each expands work over multiple inputs;gather: collects the resulting members;- utility tasks already provide a place for non-pipeline transformations;
previous_result: carries the selected/transformed result into the next stage.
So there are at least three increasingly powerful things that could be kept distinct.
1. Semantic selection
Examples:
- which composition looks best;
- which image follows the prompt best;
- which result has the best character consistency;
- which frame the user simply prefers.
For these, I think the existing flow is strong:
job A
→ agent inspects outputs
→ choose
→ keep_output
→ job B
The agent is doing something it is actually useful for, and the workflow engine does not need to grow a control-flow language.
2. Deterministic reduction
Examples:
argmax(scores)
first(candidate.score >= threshold)
min(artifact_score)
candidate[index_returned_by_previous_task]
These do not necessarily require an agent decision at all.
If this lived as a normal task/reducer, the workflow could potentially preserve the selection rule itself alongside the rest of the recipe, while still avoiding general branching semantics.
It also gives a useful place to put cheap filtering before an expensive stage:
cheap candidate generation
→ cheap evaluation
→ reduce N → 1
→ expensive video/upscale/etc.
That is the “shape” I was mainly thinking of.
3. General conditional execution
Only after that do we get to things like:
if no candidate passes:
generate another batch
else:
continue to video
That is qualitatively different. Now the DSL is not only moving data between steps; it is choosing which steps exist in the execution path.
I would probably avoid conflating this with simple selection unless a concrete workflow needs it.
A little more detail on the preflight observation
The current workflow schema lists the recognized step fields, but the step object is not closed against additional properties.
That explains why a property such as:
{
"name": "something",
"when": "previous_result:judge.pass",
"...": "..."
}
can be structurally accepted even though when is not part of the documented step vocabulary.
As a counter-check, adding additionalProperties: false only to the step schema caused the tested unknown properties and the relase_pipeline typo to fail validation while the normal baseline still passed. I do not take that as evidence that strict mode is necessarily the right policy; it just shows that the distinction is mechanically straightforward if a closed vocabulary is desired.
There is also some useful precedent inside the project itself: other layers already distinguish unknown or unused inputs rather than silently treating everything as meaningful. That is why a warning feels like a particularly low-risk option here.
A warning also has a nice property for agents: it can say something concrete such as:
Step "video": unknown property "when"; it will not affect execution.
That makes the contract much clearer than either silently accepting it or introducing an immediate compatibility-breaking error.
The ComfyUI issue I mentioned above is useful mainly because it demonstrates the operational failure mode. Their reported problem was not “the program crashed”; it was almost the opposite — the workflow looked valid enough to run, the expensive work happened, but an intended input had silently had no effect.
That seems like exactly the sort of thing a strong preflight layer is well positioned to prevent.
If general branching eventually becomes useful
If you ever do decide that the workflow itself should contain real conditions, I think the difficult part is less the syntax of when than the semantics around skipped steps.
For example:
A → maybe skipped
B → previous_result:A
What does B receive if A was skipped?
Some possibilities are:
- the reference is absent and
B also skips;
- the reference is absent and validation/runtime fails;
B supplies a default;- a fallback producer is selected;
- skipped steps produce an explicit sentinel value.
General workflow engines have to define this fairly carefully. For example, Argo’s variable/output rules explicitly deal with outputs from skipped/omitted producers rather than treating a condition as only a boolean attached to a step.
For this project there would also be questions around things that are already useful features:
- what constitutes the cache key after a branch;
- whether a skipped step has any cache/manifest record;
- whether the branch decision itself is recorded for replay;
- how
previous_result: behaves across skipped producers;
- whether changing only the selector/condition can reuse upstream cached candidates.
Retry has a similar distinction.
A technical retry:
request failed transiently
→ retry with bounded backoff
is different from a quality retry:
generation succeeded
→ nothing passed the quality threshold
→ spend more GPU time generating another batch
The second one is really an optimization/search policy, not just failure recovery. If it ever becomes part of the workflow language, I think it would be worth keeping those concepts separate.
None of this is an argument that general branching is needed now — actually, it is part of why the smaller gather → deterministic reducer shape looks attractive to me as a first boundary.
So if I had to reduce this to two concrete pieces of feedback:
-
A possible missing shape: deterministic N→1 selection/reduction between candidate generation and an expensive downstream stage, while leaving subjective “best” decisions with the agent.
-
A small preflight hardening opportunity: make unknown step-level properties visible somehow — warning, strict mode, or hard error depending on the intended extensibility of the workflow format — so a plausible agent-invented field cannot silently look meaningful.
Both seem compatible with the existing agent-first design rather than requiring it to become a general-purpose workflow engine.
