query2e: remove chat-model framing from LLM output

[radinhamidi]

Closes #42

Summary

Chat-tuned models wrap the Query2E keyword list in introductory sentences, markdown headings, category labels and <think> reasoning traces, which ended up in the expanded query. Query2E now cleans the LLM output before keyword parsing. The prompts are unchanged.

Changes

• querygym/methods/query2e.py: new clean_output param (default True). Removes <think> traces, lead-in/closing sentences, markdown headings, rules, code fences, category labels and term descriptions; keeps keyword lists, table cells, HTML/JSON lists; truncates extra Query:/Keywords: pairs in few-shot mode. Falls back to the raw output only when it is itself a plain keyword list. Per-query metadata: clean_output, unclosed_think, truncated_at_query, cleanup_fallback, cleanup_empty. clean_output=False parses the raw output exactly as before. Method version 2.0.
• querygym/core/utils.py: strip_think_trace() helper.
• querygym/core/base.py: BaseReformulator.effective_params() — the JSON-serializable method params that identify a run. Query2E records clean_output: true when enabled, so cleaned runs get their own params_hash; other methods' recorded params are unchanged.
• querygym/cli.py: --clean-output/--no-clean-output flag.
• examples/querygym_pyserini/: reformulation metadata records effective_params(); run summaries keep recorded empty params; when the reformulate step is skipped, the recorded configuration of the existing reformulated queries is reused (method/model must match).
• Docs: methods reference, API reference, reformulation guide, CLI guide, changelog.

Testing

• tests/test_methods_query2e.py: cleanup cases across output styles, think traces, few-shot truncation, fallback/empty handling, effective_params.
• reproducibility/tests/test_repro_schema.py: with clean_output=False, committed Query2E runs reproduce their recorded params_hash; the default yields a different hash. Empty recorded method params are preserved in run summaries.
• clean_output=False output and metadata are identical to main.
• Full suite: 97 passed.

🤖 Generated with Claude Code