로컬 모델의 Pi 답변 품질을 개선한 방법: 평가자의 이유 분석, 하네스 수정, 도구 결과에 가이드라인 추가
What moved a local model's answer quality in Pi: read the judge's reasons, fix the harness, and put guidance in tool results
핵심 요약
로컬 모델을 활용한 에이전트 작업에서 하네스 최적화와 도구 결과 가이드라인 추가를 통해 Claude 수준의 성능을 달성한 경험 공유.
- 평가자 피드백 분석 — 점수뿐만 아니라 구체적인 이유를 파악하여 모델의 불필요한 동작을 수정함
- 하네스 최적화 — 모델의 불필요한 파일 생성 등 하네스로 인해 발생하는 성능 저하 요인을 제거함
- 도구 결과 가이드라인 — 도구 결과 끝에 답변 형식을 명시하여 모델의 출력 품질을 크게 향상함
- 휴리스틱 검사 개선 — 모호한 텍스트 기반 검사 대신 허용 목록과 금지어 목록을 활용하여 정확도를 높임
I spent a day measuring a Pi seat on a local model (Qwen3.8 Flash-Next, 4-bit MLX mixed 4/8) against Claude Code (Sonnet 5.5) on 12 routine agent tasks, with blind pairwise judging in both orders. The local seat went from 29/36 to 36/36 passes, and judged clarity from about 3.7 to 4.5, level with Sonnet. Most of the gain came from the harness, not the model. Lessons that should apply to any Pi seat:
1. Read the judge's one-line reasons, not just the scores. Each real fix came from a sentence like "pads its answer with internal bookkeeping" or "adds a preamble when only the command was asked for". Scores told me that it was worse; reasons told me why.
2. Look for losses the harness causes. My seat injected a reminder every turn to record measured numbers in a notes file, even when no notes file existed. The model created the file in folders it was only asked about, and talked about it in replies. Only injecting the reminder when the file already exists removed the biggest clarity loss.
3. One hidden retry at agent_before_settle**.** A small extension checks the final reply before the turn ends. At most once per request, if the reply is the model's own reasoning instead of an answer, bookkeeping instead of an answer, or text around something that was asked to be "only the command", it adds a hidden custom message and continues. Two things mattered:
-
canContinueis false at that hook whenever the last message is the assistant's; the entry you add is what allows the next request, so don't skip on it. -
Steer and follow-up messages typed mid-run arrive as
inputevents withstreamingBehaviorset. Don't reset per-request state on them.
4. Put reply guidance in the tool result. The model writes its answer right after reading the tool result, so that's where guidance lands. Ending my typed drafter tool's result with "for the reply: the commit hash and which lane ran, what the commit changes, which checks did not run" moved that task from clearly behind Claude to slightly ahead.
