Skip to content

fix: add missing valid enum values to ground truth in live_simple and live_parallel#1329

Open
octo-patch wants to merge 1 commit intoShishirPatil:mainfrom
octo-patch:fix/issue-1314-ground-truth-enum-values
Open

fix: add missing valid enum values to ground truth in live_simple and live_parallel#1329
octo-patch wants to merge 1 commit intoShishirPatil:mainfrom
octo-patch:fix/issue-1314-ground-truth-enum-values

Conversation

@octo-patch
Copy link
Copy Markdown

Fixes #1314

Problem

9 ground truth entries in live_simple and live_parallel categories reject valid model outputs that are allowed by the function schema:

cmd_controller.executeunit param (4 entries in live_simple)

Entries: live_simple_141-94-0, live_simple_143-95-0, live_simple_156-95-13, live_simple_159-95-16

The unit parameter has enum: ["seconds", "milliseconds"] and default: "N/A". The ground truth only accepted "" or "N/A", rejecting "seconds" and "milliseconds" which are the valid enum values defined in the schema.

get_current_weather / get_snow_reportunit param (3 entries in live_parallel)

Entries: live_parallel_3-0-3, live_parallel_6-3-0, live_parallel_13-9-0

The unit parameter has enum: ["celsius", "fahrenheit"] and default: "fahrenheit". The ground truth only accepted "" or "fahrenheit", rejecting "celsius" — a valid enum value. Two of the three entries query locations in Celsius-speaking regions (Mexico in Spanish, France), making "celsius" a particularly reasonable choice.

Solution

  • For cmd_controller.execute entries: added "seconds" and "milliseconds" to accepted values in ground truth
  • For weather entries: added "celsius" to accepted values in ground truth

This is strictly additive — previously accepted values are unchanged, only missing valid enum values are added.

Testing

Verified the updated ground truth entries by re-reading the modified JSONL files and confirming all changes match the issue description.

… live_parallel (fixes ShishirPatil#1314)

For cmd_controller.execute entries (live_simple_141, 143, 156, 159):
- unit enum has ["seconds", "milliseconds"] but ground truth only accepted ""/"N/A"
- Added "seconds" and "milliseconds" as valid accepted values

For get_current_weather/get_snow_report entries (live_parallel_3, 6, 13):
- unit enum has ["celsius", "fahrenheit"] but ground truth only accepted ""/"fahrenheit"
- Added "celsius" as a valid accepted value since it is in the enum

Co-Authored-By: Octopus <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BFCL] Ground truth rejects valid enum values in live_parallel and live_simple

1 participant