Experimental setup
The study is organised in phases, each one answering a single question and feeding its best configuration to the next.
Data
| Set | Size | Use |
|---|---|---|
| Starter set | 40 instances | Early screening of inference-time technique families |
| Development set | 149 tasks | All later experiments and the comparisons reported on this site |
| Synthetic corpus | 3,948 pairs | Few-shot retrieval pool and supervised fine-tuning data |
| Eval-loss holdout | 197 examples | Internal validation split used as the proxy metric during the LoRA search (the cleaned release of the corpus has a 200-example holdout) |
The synthetic corpus has 3,748 training pairs and a 200-example holdout. The official GenSIE test set is evaluated in the published paper; the results on this site are on the development set.
Models
| Model | Checkpoint | Notes |
|---|---|---|
| Qwen3-14B | Qwen/Qwen3-14B |
Largest of the three |
| Salamandra-7B | BSC-LT/salamandra-7b-instruct |
Spanish-focused, trained at the Barcelona Supercomputing Center |
| Llama-3.2-3B | meta-llama/Llama-3.2-3B-Instruct |
Smallest of the three |
The same three models are used in every experiment, so cross-model comparisons are interpretable. The published paper also reports zero-shot results with Gemma-4-E4B.
The four experiments
- Inference-time technique families. Baseline, chain-of-thought, schema-aware reasoning and few-shot retrieval are screened on the starter set; the best family is then refined on the development set with constrained decoding, a field guide and post-processing layers.
- LoRA hyperparameter search. Three phases on the eval-loss holdout: rank and dropout, then scheduler, then prompt mode. Each phase carries its best setting forward.
- Effect of fine-tuning on F1. For each model, four cells on the development set: untuned baseline, untuned with the best zero-shot pipeline, LoRA, and full fine-tuning.
- Effect of fine-tuning on errors. Every error is classified into one of five categories, and the null-trap rate is measured separately as a probe for fabrication.
Evaluation protocol
- Metric: Flattened Schema Scoring Micro-F1, as defined by the task (see the problem).
- All runs are single-seed. Training runs use early stopping with a patience of 2 evaluations.
- Compute cost of an inference pipeline is estimated from token counts and parameter count.
Hardware and software
Training runs on the MareNostrum 5 supercomputer at the Barcelona Supercomputing Center, on NVIDIA H100 GPUs, scheduled with SLURM. Each LoRA run takes between 3 and 6 GPU hours. The software stack is described on the tech stack page.