Providers

accuracy

Accuracy provider Supported Measures:

• Accuracy(rel=ANY)@ANY

compat

Version of the compatibility measure desribed in:

@article{10.1145/3451161,
  author = {Clarke, Charles L. A. and Vtyurina, Alexandra and Smucker, Mark D.},
  title = {Assessing Top-k Preferences},
  journal = {ACM Transactions on Information Systems},
  volume = {39},
  number = {3},
  articleno = {33},
  numpages = {21},
  year = {2021},
  url = {https://doi.org/10.1145/3451161},
}

Supported Measures:

• Compat(p=ANY,normalize=ANY)

cwl_eval

cwl_eval, providing C/W/L (“cool”) framework measures.

https://github.com/ireval/cwl

@inproceedings{azzopardi2019cwl,
  author = {Azzopardi, Leif and Thomas, Paul and Moffat, Alistair},
  title = {cwl\_eval: An Evaluation Tool for Information Retrieval},
  booktitle = {SIGIR},
  year = {2019}
}

Supported Measures:

• P(rel=ANY,judged_only=False)@ANY

• RR(rel=ANY,judged_only=False)@NOT_PROVIDED

• AP(rel=ANY,judged_only=False)@NOT_PROVIDED

• RBP(rel=REQUIRED,p=ANY)@NOT_PROVIDED

• BPM(T=ANY,min_rel=ANY,max_rel=REQUIRED)@ANY

• SDCG(dcg='log2',min_rel=ANY,max_rel=REQUIRED)@REQUIRED

• NERR8(min_rel=ANY,max_rel=REQUIRED)@REQUIRED

• NERR9(min_rel=ANY,max_rel=REQUIRED)@REQUIRED

• NERR10(p=ANY,min_rel=ANY,max_rel=REQUIRED)

• NERR11(T=ANY,min_rel=ANY,max_rel=REQUIRED)

• INST(T=ANY,min_rel=ANY,max_rel=REQUIRED)

• INSQ(T=ANY,min_rel=ANY,max_rel=REQUIRED)

gdeval

gdeval

Supported Measures:

• nDCG(dcg='exp-log2',gains=NOT_PROVIDED,judged_only=False)@REQUIRED

• ERR@REQUIRED

judged

python implementation of judgment rate

Adapted from OpenNIR’s implementation: https://github.com/Georgetown-IR-Lab/OpenNIR/blob/master/onir/metrics/judged.py

Supported Measures:

• Judged@ANY

msmarco

MS MARCO’s implementation of RR

Supported Measures:

• RR(rel=ANY,judged_only=False)@ANY

pyndeval

pyndeval

Supported Measures:

• ERR_IA(rel=ANY,judged_only=ANY)@ANY

• nERR_IA(rel=ANY,judged_only=ANY)@ANY

• alpha_DCG(alpha=ANY,rel=ANY,judged_only=ANY)@ANY

• alpha_nDCG(alpha=ANY,rel=ANY,judged_only=ANY)@ANY

• NRBP(alpha=ANY,beta=ANY,rel=ANY)

• nNRBP(alpha=ANY,beta=ANY,rel=ANY)

• AP_IA(rel=ANY,judged_only=ANY)

• P_IA(rel=ANY,judged_only=ANY)@ANY

• StRecall(rel=ANY)@ANY

pytrec_eval

pytrec_eval

https://github.com/cvangysel/pytrec_eval

@inproceedings{VanGysel2018pytreceval,
    title={Pytrec\_eval: An Extremely Fast Python Interface to trec\_eval},
    author={Van Gysel, Christophe and de Rijke, Maarten},
    publisher={ACM},
    booktitle={SIGIR},
    year={2018},
}

Supported Measures:

• P(rel=ANY,judged_only=ANY)@ANY

• RR(rel=ANY,judged_only=ANY)@NOT_PROVIDED

• Rprec(rel=ANY,judged_only=ANY)

• AP(rel=ANY,judged_only=ANY)@ANY

• nDCG(dcg='log2',gains=ANY,judged_only=ANY)@ANY

• R(judged_only=ANY)@ANY

• Bpref(rel=ANY)

• NumRet(rel=ANY)

• NumQ

• NumRel(rel=1)

• SetAP(rel=ANY,judged_only=ANY)

• SetF(rel=ANY,beta=ANY,judged_only=ANY)

• SetP(rel=ANY,relative=ANY,judged_only=ANY)

• SetR(rel=ANY)

• Success(rel=ANY,judged_only=ANY)@ANY

• IPrec(judged_only=ANY)@ANY

• infAP(rel=ANY)

ranx

ranx

https://amenra.github.io/ranx/

@misc{ranx2021,
  title = {ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison},
  author = {Bassani, Elias},
  year = {2021},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/AmenRa/ranx}},
}

Supported Measures:

• P(rel=ANY,judged_only=False)@ANY

• SetP(rel=ANY,judged_only=False)

• RR(rel=ANY,judged_only=False)@NOT_PROVIDED

• Rprec(rel=ANY,judged_only=False)

• AP(rel=ANY,judged_only=False)@ANY

• nDCG(dcg=('log2', 'exp-log2'),gains=NOT_PROVIDED,judged_only=False)@ANY

• R(judged_only=False)@ANY

• SetR(rel=ANY)

• NumRet(rel=REQUIRED)

• Success(rel=ANY,judged_only=False)@REQUIRED

runtime

Supports measures that are defined at runtime via ir_measures.define() and ir_measures.define_byquery().

Supported Measures:

trectools

trectools

https://github.com/joaopalotti/trectools

@inproceedings{palotti2019,
   author = {Palotti, Joao and Scells, Harrisen and Zuccon, Guido},
   title = {TrecTools: an open-source Python library for Information Retrieval practitioners involved in TREC-like campaigns},
   series = {SIGIR'19},
   year = {2019},
   location = {Paris, France},
   publisher = {ACM}
}

Supported Measures:

• P(rel=1,judged_only=False)@ANY

• RR(rel=1,judged_only=False)@NOT_PROVIDED

• Rprec(rel=1,judged_only=False)

• AP(rel=1,judged_only=False)@ANY

• nDCG(dcg=ANY,gains=NOT_PROVIDED,judged_only=False)@ANY

• Bpref(rel=1)

• RBP(p=ANY,rel=ANY)@ANY