[BUG] flytekit v1.16.15 breaks map_task: ArrayNode references unreleased flyteidl field run_all_sub_nodes

[tradeqvest]

Flyte & Flytekit version

• flytekit: 1.16.15
• flyteidl: 1.16.4
• protobuf: 6.33.5
• Python: 3.12
• Flyte server: v1.16.3 (Helm chart)

Package	Version	Has run_all_sub_nodes?
flytekit	1.16.13	No (works)
flytekit	1.16.14	No (works)
flytekit	1.16.15	Yes (broken)
flyteidl	1.16.4 (latest 1.x)	No

Describe the bug

flytekit==1.16.15 introduced run_all_sub_nodes support for ArrayNode via flytekit#3399, but the corresponding flyteidl proto field has not been released. The proto change (flyte#6324) is still in draft and was never merged.

Any workflow using map_task fails on registration:

ValueError: Protocol message ArrayNode has no "run_all_sub_nodes" field.

Expected behavior

Root cause

In flytekit/models/core/workflow.py, the ArrayNode class was updated to reference run_all_sub_nodes:

• Line 429: run_all_sub_nodes=self._run_all_sub_nodes in to_flyte_idl()
• Line 439: run_all_sub_nodes=pb2_object.run_all_sub_nodes in from_flyte_idl()

But flyteidl==1.16.4 (the latest release satisfying flyteidl>=1.16.1,<2.0.0a0) does not have this field. The ArrayNode proto only defines fields up to bound_inputs (field number 8).

Suggested fix

Either:

1. Release a new flyteidl 1.x with the run_all_sub_nodes field on ArrayNode (merge flyte#6324)
2. Revert flytekit#3399 from the 1.16.x branch until the proto is released

Additional context to reproduce

pip install flytekit==1.16.15  # installs flyteidl==1.16.4 (latest 1.x)

from flytekit import task, workflow, map_task
from functools import partial

@task
def add_one(x: int, offset: int = 1) -> int:
    return x + offset

@workflow
def test_map() -> list[int]:
    return map_task(partial(add_one, offset=1))(x=[1, 2, 3])

pyflyte run --remote test.py test_map
# ValueError: Protocol message ArrayNode has no "run_all_sub_nodes" field.

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

It could be possible that this issue arises because flytekit==1.16.15 references the run_all_sub_nodes field in the ArrayNode proto, but the latest released flyteidl==1.16.4 does not include this field. The proto change adding run_all_sub_nodes is still in draft and not merged or released, so any attempt to use map_task with flytekit==1.16.15 fails at registration with a ValueError about the missing field. This is consistent with your findings and the root cause you described.

Though I'm not entirely certain, it might be that the only solutions are to either (1) release a new flyteidl 1.x version with the run_all_sub_nodes field after merging the relevant PR, or (2) revert the change in flytekit that references this field until the proto is available. This incompatibility is not explicitly documented in the available changelogs or RFCs, but the ArrayNode RFC and CHANGELOG-v1.9.0.md discuss ArrayNode and map_task changes, though not this specific field or issue.

Would you like a more detailed breakdown of the relevant code changes and possible workarounds?

Sources:

• ArrayNode RFC
• CHANGELOG-v1.9.0.md

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[tradeqvest]

@pvditt any insights from you regarding this issue?