DeerFlow skill-creator JSON Schema 全解从 evals 到 benchmark 的八类数据契约【免费下载链接】deer-flowAn open-source long-horizon SuperAgent harness that researches, codes, and creates. With the help of sandboxes, memories, tools, skill, subagents and message gateway, it handles different levels of tasks that could take minutes to hours.项目地址: https://gitcode.com/GitHub_Trending/de/deer-flowskills/public/skill-creator/references/schemas.md是 DeerFlow 技能创作工作流的字段级合同它把 evals、grading、metrics、timing、benchmark、comparison、analysis、history 八类 JSON 文件的字段、嵌套位置与命名红线一次性钉死。读完本篇你将能准确写出被聚合脚本、viewer 与三个子代理grader/comparator/analyzer消费的文件结构避免因字段命名差异导致 benchmark 视图出现空值或零值。一、八类 JSON 文件在 skill-creator 数据流中的位置skill-creator 的核心循环是SKILL.md 草稿 → 运行 with_skill / without_skill 两组子代理 → 打分 → 聚合 → 盲评与归因 → 反馈迭代。八类 JSON 文件分别落在流水线的不同节点且彼此存在明确的上下游关系阶段文件写入方消费方测试用例定义evals/evals.json主代理skill-creator 本体子代理 prompt、viewer版本演进工作区根history.json主代理在 Improve 模式主代理决策下一版本执行指标run-dir/outputs/metrics.json执行子代理grader、聚合脚本墙钟计时run-dir/timing.json主代理在任务通知到达时grader、聚合脚本打分输出run-dir/grading.jsongrader 子代理聚合脚本、viewer聚合基准benchmarks/ts/benchmark.jsonaggregate_benchmark.pyviewer、benchmark.md生成盲评输出grading-dir/comparison-N.jsoncomparator 子代理主代理决策归因输出grading-dir/analysis.jsonanalyzer 子代理主代理改进技能这条链路在 SKILL.md 中有完整叙述例如 Step 3 明确要求把total_tokens与duration_ms立刻写入timing.jsonStep 4 要求grade → aggregate → viewer。schemas.md 则是这些文件的字段级字典是脚本与子代理之间的共同契约。二、evals.json测试用例的入口定义位于技能目录下的evals/evals.json是 skill-creator 测试用例的唯一定义入口。schemas.md 给出的完整示例{ skill_name: example-skill, evals: [ { id: 1, prompt: Users example prompt, expected_output: Description of expected result, files: [evals/files/sample1.pdf], expectations: [ The output includes X, The skill used script Y ] } ] }字段级约束skill_name必须与 SKILL.md frontmatter 中的name字段严格一致utils.py 的parse_skill_md会解析该字段若两者不一致run_eval.py与run_loop.py会读到错误的 name。evals[].id唯一整数聚合脚本用它作为eval_id写回benchmark.json。evals[].prompt实际下发给执行子代理的任务描述。evals[].expected_output面向人类的成功描述供打分代理参考但不直接用于打分。evals[].files可选的输入文件路径列表相对于技能根目录子代理 prompt 会据此挂载文件。evals[].expectations可验证的陈述句是后续 grading 与 benchmark 的原子单位。一个容易被忽略的细节SKILL.md 建议先只写 prompt断言留到 run 进行中的 Step 2 补也就是说初版evals.json可以不含expectations字段跑起来后由主代理根据 transcript 与用户访谈补充。这条约定保证了测试集可以边跑边完善而不是被前置的断言设计卡住。三、history.jsonImprove 模式的版本演进台账history.json位于工作区根目录不是技能目录专门追踪在既有 skill 基础上迭代的场景。schemas.md 给出的示例{ started_at: 2026-01-15T10:30:00Z, skill_name: pdf, current_best: v2, iterations: [ { version: v0, parent: null, expectation_pass_rate: 0.65, grading_result: baseline, is_current_best: false }, { version: v1, parent: v0, expectation_pass_rate: 0.75, grading_result: won, is_current_best: false }, { version: v2, parent: v1, expectation_pass_rate: 0.85, grading_result: won, is_current_best: true } ] }字段要点started_atISO 时间戳标记改进循环的起点。skill_name正在改进的技能名与 SKILL.md 一致。current_best当前最佳版本的标识符如v2主代理据此决定下一版基于哪个 parent 派生。iterations[].versionv0、v1、… 单调递增。iterations[].parent该版本基于哪个版本派生v0为null。iterations[].expectation_pass_rate本轮 grading 得到的 pass rate是版本比较的量化依据。iterations[].grading_result四值枚举——baseline/won/lost/tiebaseline只在v0出现。iterations[].is_current_best布尔标记全文件有且仅有一处为true与顶层current_best保持同步。这份文件的用途是让主代理在多轮改—跑—比循环中快速回答现在应该以哪一版为 parent 继续改。当grading_result连续为lost时主代理应回滚到current_best并改变改进策略而不是继续线性推进。四、grading.jsongrader 子代理的输出契约run-dir/grading.json由 grader 子代理产出agents/grader.md 定义了完整的 8 步流程读 transcript → 检查输出 → 逐条断言 → 抽取 claims → 读 user_notes → 批判 evals → 写结果 → 读 metrics/timing。schemas.md 中的示例{ expectations: [ { text: The output includes the name John Smith, passed: true, evidence: Found in transcript Step 3: Extracted names: John Smith, Sarah Johnson }, { text: The spreadsheet has a SUM formula in cell B10, passed: false, evidence: No spreadsheet was created. The output was a text file. } ], summary: { passed: 2, failed: 1, total: 3, pass_rate: 0.67 }, execution_metrics: { tool_calls: { Read: 5, Write: 2, Bash: 8 }, total_tool_calls: 15, total_steps: 6, errors_encountered: 0, output_chars: 12450, transcript_chars: 3200 }, timing: { executor_duration_seconds: 165.0, grader_duration_seconds: 26.0, total_duration_seconds: 191.0 }, claims: [ { claim: The form has 12 fillable fields, type: factual, verified: true, evidence: Counted 12 fields in field_info.json } ], user_notes_summary: { uncertainties: [Used 2023 data, may be stale], needs_review: [], workarounds: [Fell back to text overlay for non-fillable fields] }, eval_feedback: { suggestions: [ { assertion: The output includes the name John Smith, reason: A hallucinated document that mentions the name would also pass } ], overall: Assertions check presence but not correctness. } }字段红线聚合脚本与 viewer 都强依赖expectations[]的三字段必须叫text/passed/evidence。SKILL.md 在 Step 4.1 显式强调不能用name/met/details等变体——viewer 依赖这些精确字段名。aggregate_benchmark.py 的load_run_results在读取时也会逐条校验text与passed是否存在缺失会打印 warning。summary.pass_rate是聚合脚本pass_rate的唯一来源见load_run_results中grading.get(summary, {}).get(pass_rate, 0.0)。execution_metrics从执行子代理的metrics.json原样透传grader 只做复制不做改写。timing从timing.json透传executor_duration_seconds与grader_duration_seconds之和应近似等于total_duration_seconds。claims[].type三值枚举factual/process/qualityverified是布尔。user_notes_summary的三个子数组uncertainties/needs_review/workarounds会被聚合脚本合并成 benchmark 的notes见aggregate_benchmark.pyL164-L169。eval_feedback是可选字段仅在 grader 认为 evals 本身存在问题时出现用于反哺evals.json的下一版。五、metrics.json执行子代理的运行指标run-dir/outputs/metrics.json由执行子代理在任务结束时产出是 grader 与聚合脚本共同的数据源。schemas.md 示例{ tool_calls: { Read: 5, Write: 2, Bash: 8, Edit: 1, Glob: 2, Grep: 0 }, total_tool_calls: 18, total_steps: 6, files_created: [filled_form.pdf, field_values.json], errors_encountered: 0, output_chars: 12450, transcript_chars: 3200 }字段语义tool_calls按工具名分桶的调用次数键是工具名Read、Write、Bash、Edit、Glob、Grep等。total_tool_calls所有工具调用之和等于sum(tool_calls.values())。total_steps主执行步骤数区别于子步骤。files_created执行过程中产出的文件相对路径列表。errors_encountered执行期间捕获到的错误数工具报错、脚本非零退出等。output_chars所有输出文件的字符数总和是token 成本的代理指标grader 文档明确注明 proxy for tokens。transcript_chars完整 transcript 的字符数用于诊断执行是否啰嗦。metrics.json与grading.json的execution_metrics字段应当一致——grader 的职责是透传而非重算若二者不一致通常意味着 grader 实现有 bug。六、timing.json墙钟计时与任务通知窗口run-dir/timing.json是主代理在子代理任务通知到达的瞬间写下的文件schemas.md 用加粗强调了这一点How to capture:When a subagent task completes, the task notification includestotal_tokensandduration_ms. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.示例{ total_tokens: 84852, duration_ms: 23332, total_duration_seconds: 23.3, executor_start: 2026-01-15T10:30:00Z, executor_end: 2026-01-15T10:32:45Z, executor_duration_seconds: 165.0, grader_start: 2026-01-15T10:32:46Z, grader_end: 2026-01-15T10:33:12Z, grader_duration_seconds: 26.0 }字段说明total_tokens/duration_ms直接从任务通知抄录只此一次、事后不可恢复。total_duration_secondsduration_ms / 1000的近似供聚合脚本直接读取。executor_*/grader_*可选的细粒度分段用于诊断执行阶段花了多久、打分阶段花了多久。顶层三个_duration_seconds之间应满足executor_duration_seconds grader_duration_seconds ≈ total_duration_seconds若包含其他阶段则允许差异。aggregate_benchmark.py 的取数顺序值得注意先看grading.json内的timing.total_duration_seconds若为 0回退读同级timing.json的total_duration_seconds并从中读取total_tokens作为tokens字段。这条回退逻辑说明只要 timing.json 存在且字段完整即使 grading.json 没透传 timing聚合也能拿到时间。七、benchmark.json聚合脚本的输出与 viewer 的强契约benchmarks/timestamp/benchmark.json由 aggregate_benchmark.py 生成是整条流水线的汇总文件。schemas.md 示例节选{ metadata: { skill_name: pdf, skill_path: /path/to/pdf, executor_model: claude-sonnet-4-20250514, analyzer_model: most-capable-model, timestamp: 2026-01-15T10:30:00Z, evals_run: [1, 2, 3], runs_per_configuration: 3 }, runs: [ { eval_id: 1, eval_name: Ocean, configuration: with_skill, run_number: 1, result: { pass_rate: 0.85, passed: 6, failed: 1, total: 7, time_seconds: 42.5, tokens: 3800, tool_calls: 18, errors: 0 }, expectations: [ {text: ..., passed: true, evidence: ...} ], notes: [Used 2023 data, may be stale] } ], run_summary: { with_skill: { pass_rate: {mean: 0.85, stddev: 0.05, min: 0.80, max: 0.90}, time_seconds: {mean: 45.0, stddev: 12.0, min: 32.0, max: 58.0}, tokens: {mean: 3800, stddev: 400, min: 3200, max: 4100} }, without_skill: { ...: ... }, delta: { pass_rate: 0.50, time_seconds: 13.0, tokens: 1700 } }, notes: [ Assertion Output is a PDF file passes 100% in both configurations, Eval 3 shows high variance (50% ± 40%), Without-skill runs consistently fail on table extraction expectations ] }schemas.md 特别用Important段落强调了字段名红线The viewer reads these field names exactly. Usingconfiginstead ofconfiguration, or puttingpass_rateat the top level of a run instead of nested underresult, will cause the viewer to show empty/zero values.从 aggregate_benchmark.py 源码可以印证每一条约束configuration字段脚本generate_benchmarkL240 写死configuration: config其中config来自eval_dir下的目录名with_skill/without_skill/old_skill/new_skill均可。目录名即字段值viewer 按该字符串做分组与配色。result嵌套pass_rate/passed/failed/total/time_seconds/tokens/tool_calls/errors全部必须嵌套在result下脚本 L242-L251 显式构造了这个嵌套对象。run_summary的三指标pass_rate/time_seconds/tokens各自包含mean/stddev/min/max四字段由calculate_stats计算标准差使用n-1无偏估计L53-L55n1时stddev为 0.0。delta是三个字符串如0.50、13.0、1700由脚本 L218-L222 格式化生成取configs[0]与configs[1]的均值差顺序敏感——SKILL.md 明确要求with_skill 版本放在 baseline 之前就是为了保证 delta 是技能收益而非基线减去技能。notes数组由两部分组成聚合阶段为[]随后由 analyzer.md 的Analyzing Benchmark Results章节补全输出为字符串数组 JSON。eval_name字段如Ocean用于 viewer 的分节标题来自eval_metadata.json的eval_name缺失时聚合脚本回退为整数eval_id。八、comparison.json盲评对比的结构grading-dir/comparison-N.json由 comparator 子代理产出agents/comparator.md 定义了 7 步流程读两输出 → 理解任务 → 生成 rubric → 打分 → 断言核对 → 决定胜者 → 写 JSON。schemas.md 示例{ winner: A, reasoning: Output A provides a complete solution with proper formatting ..., rubric: { A: { content: { correctness: 5, completeness: 5, accuracy: 4 }, structure: { organization: 4, formatting: 5, usability: 4 }, content_score: 4.7, structure_score: 4.3, overall_score: 9.0 }, B: { ...: ... } }, output_quality: { A: { score: 9, strengths: [Complete solution, Well-formatted, All fields present], weaknesses: [Minor style inconsistency in header] }, B: { ...: ... } }, expectation_results: { A: { passed: 4, total: 5, pass_rate: 0.80, details: [ {text: Output includes name, passed: true} ] }, B: { ...: ... } } }字段要点winnerA/B/TIEreasoning必须解释为何该胜者胜。rubric.side内部分两个维度contentcorrectness/completeness/accuracy1-5 整数分。structureorganization/formatting/usability1-5 整数分。content_score/structure_score各自维度的均值1-5。overall_score1-10 缩放后的综合分不是 content/structure 两维度的均值而是加权后映射到 1-10。output_quality.sidescore1-10应与rubric.overall_score保持一致comparator.md 明确 should match rubric overall_score。strengths/weaknesses可读的具体条目而非抽象评价。expectation_results仅在提供 expectations 时出现comparator.md 明确 If no expectations were provided, omit theexpectation_resultsfield entirely. 每个 side 内含passed/total/pass_rate/details[]details元素只含text与passed两字段比 grading.json 少了evidence。盲评的关键约束在 comparator.md 的 GuidelinesStay blind: DO NOT try to infer which skill produced which output——comparison.json 本身不包含任何 skill 路径胜者到 skill 的映射由下游 analysis.json 补上。九、analysis.json事后归因与揭示结构grading-dir/analysis.json由 analyzer 子代理产出agents/analyzer.md 定义了 8 步流程。它的职责是unblind——拿到 comparison 的 winner 后读双方 SKILL.md 与 transcript回答赢家为何赢、输家如何改进。schemas.md 示例{ comparison_summary: { winner: A, winner_skill: path/to/winner/skill, loser_skill: path/to/loser/skill, comparator_reasoning: Brief summary of why comparator chose winner }, winner_strengths: [ Clear step-by-step instructions for handling multi-page documents, Included validation script that caught formatting errors ], loser_weaknesses: [ Vague instruction process the document appropriately led to inconsistent behavior, No script for validation, agent had to improvise ], instruction_following: { winner: { score: 9, issues: [Minor: skipped optional logging step] }, loser: { score: 6, issues: [ Did not use the skills formatting template, Invented own approach instead of following step 3 ] } }, improvement_suggestions: [ { priority: high, category: instructions, suggestion: Replace process the document appropriately with explicit steps, expected_impact: Would eliminate ambiguity that caused inconsistent behavior } ], transcript_insights: { winner_execution_pattern: Read skill - Followed 5-step process - Used validation script, loser_execution_pattern: Read skill - Unclear on approach - Tried 3 different methods } }字段红线comparison_summary内winner_skill/loser_skill是技能路径不是版本号与 comparison.json 的 A/B 抽象解耦。instruction_following.side.score1-10issues[]列出具体扣分点。improvement_suggestions[]每项四字段priorityhigh/medium/lowanalyzer.md 明确定义三档语义。category六值枚举——instructions/tools/examples/error_handling/structure/referencesanalyzer.md 附了完整表格。suggestion可执行的具体改动描述。expected_impact预期影响让主代理能判断值不值得改。transcript_insights两个字符串字段用-串联执行模式便于主代理快速对比执行轨迹差异。analyzer.md 中还有一个Analyzing Benchmark Results分支不是同一份 JSON而是输出字符串数组 JSON 到benchmark.json.notes用于聚合后的模式发现——这个分支与 analysis.json 是并列但独立的两条产出路径前者针对单对 A/B后者针对整轮 benchmark。十、字段命名红线与可复现契约总表把上面各节的硬依赖汇总成一张速查表方便在写或改任何 skill-creator 相关脚本时对照文件关键字段viewer/脚本按名读取出现位置evals.jsonevals[].id/prompt/files/expectations顶层evals数组history.jsoncurrent_best/iterations[].version/parent/grading_result/is_current_best顶层 iterations[]grading.jsonexpectations[].text/passed/evidencesummary.pass_rateexecution_metricstiminguser_notes_summaryeval_feedback顶层各键metrics.jsontool_calls/total_tool_calls/total_steps/output_chars/transcript_chars顶层timing.jsontotal_tokens/duration_ms/total_duration_seconds顶层benchmark.jsonruns[].configuration非configruns[].result.pass_rate非顶层run_summary.cfg.metric.{mean,stddev,min,max}delta.{pass_rate,time_seconds,tokens}顶层 嵌套comparison.jsonwinner/rubric.side.{content,structure,content_score,structure_score,overall_score}/output_quality.side.score/expectation_results可选顶层analysis.jsoncomparison_summary/instruction_following.side.score/improvement_suggestions[].{priority,category,suggestion,expected_impact}顶层三条最容易踩的坑schemas.md 显式或 SKILL.md 显式强调grading.json.expectations[]三字段必须是text/passed/evidence不能用name/met/detailsaggregate_benchmark.py L159 会逐条校验并 warning。benchmark.json.runs[]的configuration是目录名原样不是configresult.pass_rate必须嵌套在result下不能提到 run 顶层。timing.json的total_tokens与duration_ms只在任务通知的瞬间可获得事后不可补——必须到达即写。十一、如何验证自己的实现符合契约三条可执行的最小验证路径都不需要修改 skill-creator 源码单元自检把一份手工构造的grading.json放到iteration-N/eval-0/with_skill/run-1/下运行 aggregate_benchmark.pypython -m scripts.aggregate_benchmark workspace/iteration-N --skill-name name若 stdout 出现 Warning: expectation in ... missing required fields (text, passed, evidence)说明 grading.json 字段名不对。viewer 校验用 generate_review.py 打开生成的benchmark.json若Benchmark tab 的数值列全为 0/空几乎一定是configuration字段名或result嵌套层级出了问题对应 schemas.md 的 Important 段落。触发面校验evals.json的skill_name与 SKILL.md frontmattername不一致时run_eval.py 生成的 command 文件名会用错 name导致claude -p侧的available_skills命中失败——可通过在--verbose输出中比对Evaluating:行与SKILL.md的 description 是否一致来快速发现。十二、小结schemas.md 表面上是一份JSON 字段字典实质是 DeerFlow skill-creator 数据流的接口层它把主代理、执行子代理、grader、comparator、analyzer 五个角色以及聚合脚本、viewer 两个消费端统一钉在同一组字段名上。理解这份 schema 的最佳方式不是逐字段背诵而是沿着evals.json → metrics/timing → grading → benchmark → comparison/analysis → history这条数据流走一遍看每个文件的每一处写入方与读取方——这正是 SKILL.md 的 Step 1–5 与 agents/ 目录中三份代理文档共同描述的那条闭环。【免费下载链接】deer-flowAn open-source long-horizon SuperAgent harness that researches, codes, and creates. With the help of sandboxes, memories, tools, skill, subagents and message gateway, it handles different levels of tasks that could take minutes to hours.项目地址: https://gitcode.com/GitHub_Trending/de/deer-flow创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考