Skip to content

Results

Numbers and observations only; interpretation is on What I learned. All results are on the GenSIE development set (149 tasks), single-seed.

Best zero-shot pipeline: progression on Qwen3-14B

Pipeline F1
Baseline (zero-shot) 0.6577
E021 (1-shot) 0.6994
E025 (3-shot, schema-similarity) 0.7155
E211 (E025 + grounding post-processing) 0.7426
E222 (E025 + full post-processing, including the hallucination filter) 0.7840
E225 (enriched selector, synthetic library, behavioural addenda) 0.7910
E226 (E225 + field guide + source cleaning) 0.7999

Across the whole post-processing stack, the hallucination error count fell from 84 (E025, without post-processing) to 21 (E222). A variant of E226 with a 40% shorter prompt (E227) scored 0.7895.

Best result per regime and model

Model Untuned baseline Best zero-shot Best LoRA Best full fine-tuning
Qwen3-14B 0.6577 0.7999 0.6977 0.7054
Llama-3.2-3B 0.6703 0.7459 0.7449 0.7481
Salamandra-7B 0.4843 0.6276 0.6406 0.6193

Best value per model in bold. The tuned Qwen and Llama models were evaluated only with the baseline and 1-shot prompts: the richest pipelines (E225, E226) were not run on them, and the Qwen LoRA run with E226 was abandoned because of prompt length and latency. The table therefore does not show how much training adds on top of the best pipeline for those two models. The single highest F1 across all configurations is Qwen3-14B zero-shot (0.7999). On Llama the best zero-shot pipeline (0.7459) and full fine-tuning (0.7481) are essentially tied. On Salamandra the best result is LoRA combined with an enriched few-shot prompt.

Hallucination on nullable fields

Rates over the 505 nullable-field opportunities of the development set, with the plain baseline prompt (no few-shot examples, no post-processing).

Model Untuned null-trap rate Best tuned variant
Qwen3-14B 14.65% 12.08% (full fine-tuning; LoRA 12.48%)
Salamandra-7B 24.55% 7.33% (full fine-tuning; LoRA 14.26%)
Llama-3.2-3B 25.15% 6.34% (LoRA; full fine-tuning 12.67%)

The rate fell for every model under training. All rates were measured with the plain baseline prompt, not with the best pipelines.

The search converged on the same configuration for the three models: rank 64, alpha 128, dropout 0.10, learning rate 5e-5. Going from rank 64 to rank 128 lowered eval-loss by at most 0.003 while the train–eval gap kept growing. In-context tuning gave the lowest eval-loss on every model:

Model Default prompt eval-loss In-context tuning eval-loss
Qwen3-14B 0.5698 0.3569
Salamandra-7B 0.5078 0.3829
Llama-3.2-3B 0.671 0.4248