Skip to content

The problem

Schema-guided extraction in Spanish

An instance of the task is a triple: a Spanish source text, a natural-language instruction, and a JSON Schema that defines what to extract. The schema is only seen at inference time, so a system cannot memorise entity types: it has to read field names, types and descriptions and act on them.

A valid output must satisfy two constraints at once:

  1. Structural validity: the output is a JSON object that conforms to the schema.
  2. Source faithfulness: every non-null value is supported by the text, and every field the text does not support is returned as null.

Below is an illustrative instance written for this page, in the format of the synthetic corpus (schema abridged); it is not a corpus record. Note that the average flow is not in the text, so the correct answer for that field is null.

{
  "input_text": "El río Arga es uno de los principales ríos de Navarra. Nace en el área de Quinto Real, en los Pirineos navarros, y atraviesa Pamplona y Puente la Reina antes de desembocar en el río Aragón. Tiene una longitud de 145 km.",
  "instruction": "Extrae el perfil estructurado del río descrito: su nombre, el río en el que desemboca, su longitud en kilómetros, la zona donde nace, las localidades que atraviesa y su caudal medio.",
  "target_schema": "RiverProfile: name, tributary_of, length_km (integer), source_range, passes_through (array), avg_flow_m3s (number or null)",
  "output": {
    "name": "Arga",
    "tributary_of": "Aragón",
    "length_km": 145,
    "source_range": "Quinto Real",
    "passes_through": ["Pamplona", "Puente la Reina"],
    "avg_flow_m3s": null
  }
}

Null-traps

Fields that the text does not answer are called null-traps. A model that invents a plausible value for them is hallucinating, and the evaluation penalises it. This makes the task a test of restraint as much as of extraction.

The GenSIE 2026 shared task

GenSIE (General-purpose Schema-guided Information Extraction) was a shared task at IberLEF 2026. Two rules shaped the work:

  • No-Training policy. Fine-tuning the model weights (LoRA, QLoRA, full fine-tuning) was prohibited. Only inference-time techniques were allowed: prompting, retrieval, constrained decoding and few-shot examples.
  • Several small open models. Systems were evaluated on several open-weight models below 14B parameters from different families, some of them held out until the results, so tuning a pipeline to a single model was discouraged.

Roughly half of the private test schemas were entirely new.

The metric

The primary metric is Flattened Schema Scoring: gold and system outputs are flattened into key–value sets, each shared key receives a similarity score between 0 and 1 (exact match for rigid types such as enums and numbers, an embedding and lexical blend for free text), and the final score is a Micro-F1:

F1 = 2 · TPS / (|G| + |S|), where TPS is the sum of the similarities of the shared keys, and |G| and |S| are the sizes of the gold and system key sets.

The GenSIE leaderboard aggregates results as the fraction of the baseline-to-perfect gap each system closes, averaged over the evaluation models. This site reports raw Micro-F1 per model.