API Reference¶

Metric Calculation¶

The following top-level functions calculate measures using any available provider.

ir_measures.calc(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]], run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → CalcResults¶: Returns aggregated and per-query results for these measures, qrels, and run.

ir_measures.calc_aggregate(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]], run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → Dict[Measure, float | int]¶: Returns aggregated measure values for these measures, qrels, and run.

ir_measures.iter_calc(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]], run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → Iterator[Metric]¶: Yields per-topic metrics for these measures, qrels, and run.

ir_measures.evaluator(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]]) → Evaluator¶: Returns an Evaluator for these measures and qrels, which can efficiently process multiple runs.

Base Classes¶

class ir_measures.providers.Provider¶

The base class for all measure providers (e.g., pytrec_eval, gdeval, etc.).

A Provider implements the calculation logic for one or more Measure (e.g., nDCG, P, etc.).

calc(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]], run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → CalcResults¶: Returns aggregated and per-query results for these measures, qrels, and run.

calc_aggregate(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]], run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → Dict[Measure, float | int]¶: Returns aggregated measure values for these measures, qrels, and run.

evaluator(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]]) → Evaluator¶: Returns an Evaluator for these measures and qrels, which can efficiently process multiple runs.

iter_calc(measures: Iterable[Measure], qrels: Iterable[Qrel] | pandas.DataFrame | Dict[str, Dict[str, int]], run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → Iterator[Metric]¶: Yields per-topic metrics for these measures, qrels, and run.

class ir_measures.providers.Evaluator(measures: Iterable[Measure], qrel_qids: Iterable[str])¶

The base class for scoring runs for a given set of measures and qrels. Returned from Provider.evaluator().

calc(run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → CalcResults¶: Returns aggregated and per-query results for this run.

calc_aggregate(run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → Dict[Measure, float | int]¶: Returns aggregated measure values for this run.

iter_calc(run: Iterable[ScoredDoc] | pandas.DataFrame | Dict[str, Dict[str, float]]) → Iterator[Metric]¶: Yields per-topic metrics this run.

class ir_measures.measures.Measure(**params)¶

Parsing¶

ir_measures.parse_measure(measure: str) → Measure¶

ir_measures.parse_trec_measure(measure: str) → List[Measure]¶

ir_measures.read_trec_qrels(file)¶

ir_measures.read_trec_run(file)¶

Custom Measures¶

See here for an example of using custom measures.

ir_measures.define(impl: Callable[[pd.DataFrame, pd.DataFrame], Iterable[Tuple[str, float]]], name: str | None = None, support_cutoff: bool = True)¶

Defines a new custom measure from a user-specified function that is is provided all queries at once.

impl is a function that accepts (qrels, run) and returns an iterable of (qid, score) tuples.

Most of the time, it is probably easier to use define_byquery(), since it operates with one query at a time.

Parameters:

impl – A function that takes two pandas DataFrames (qrels and run) and returns an iterable of (qid, score) tuples.
name – The name of the measure (optional)
support_cutoff – Whether the measure supports a cutoff parameter, which reduces the results in run.

ir_measures.define_byquery(impl: Callable[[pd.DataFrame, pd.DataFrame], float], name: str | None = None, support_cutoff: bool = True)¶

Defines a new custom measure from a user-specified function that is called once per query.

impl is a function that accepts (qrels, run) and is called once per query, returning a float value each time for the specific query.

Parameters:

impl – A function that takes two pandas DataFrames (qrels and run) and returns a float.
name – The name of the measure (optional)
support_cutoff – Whether the measure supports a cutoff parameter, which reduces the results in run.

Data Classes¶

class ir_measures.Metric(query_id, measure, value)¶

class ir_measures.Qrel(query_id, doc_id, relevance, iteration)¶

class ir_measures.ScoredDoc(query_id, doc_id, score)¶

class ir_measures.CalcResults(aggregated, per_query)¶