← Go back

GERBIL

Production-ready project (September 2014 - Ongoing)

General Entity Annotation Benchmark Framework

About the project

GERBIL is a general Linked Data benchmarking system (formerly used for entity annotation systems based on the BAT-Framework). GERBIL offers an easy-to-use web-based platform for the agile comparison of annotators using multiple datasets and uniform measuring approaches. To add a tool to GERBIL, all the end user has to do is to provide a URL to a REST interface to its tool which abides by a given specification. The integration and benchmarking of the tool against user-specified datasets is then carried out automatically by the GERBIL platform.

Available Annotators

BAT-FRAMEWORKGERBIL 1.0.0GERBIL1.2.5ExperimentPaper
AGDISTIS(✔)D2KBLink
AIDAA2KBLink
BabelyA2KBLink
CETUSOKE Task 2Link
CETUS (FOX)OKE Task 2Link
DBpedia SpotlightA2KBLink
DexterA2KBLink
DoSeR (*)D2KBLink
entityclassifier.eu NERA2KBLink
FOXOKE Task 1Link
FREDOKE Task 1Link
FREME NEROKE Task 1Link
KEAA2KBLink
NERD-MLA2KBLink
NERFGUND2KBLink
OpenTapiocaA2KBLink
PBOHD2KBLink
TagMe 2A2KBLink
WATA2KBLink
xLisaA2KBLink

(*) Annotator isn't available any more

Supported Experiments for each Annotator

The following table lists the annotators that are currently available and the experiment types they support. Note that some of the A2KB annotators support the D2KB experiment by offering an own API method. Other A2KB annotators can be chosen for a D2KB experiment as well as described in the wiki. However, since the comparison might not be fair, we marked these annotators with (✔) in the table. The same is done for Entity Typing.

A2KB, C2KB
ENTITY RECOGNITION
D2KBENTITY
TYPING
OKE TASK 1OKE TASK 2RT2KBRE
AGDISTIS
AIDA(✔)
Babely
CETUS
CETUS (FOX)
DBpedia Spotlight
Dexter(✔)
DoSeR (*)
entityclassifier.eu NER(✔)
FOX(✔)(✔)
FRED(✔)(✔)
FREME NER
KEA
NERD-ML(✔)
NERFGUN
OpenTapioca
PBOH
TagMe 2(✔)
WAT
xLisa(✔)

(*) Annotator isn't available any more

Available Datasets

The following table lists the datasets that are currently available and the experiment types they support.

A2KB, C2KB, D2KB
ENTITY RECOGNITION
ENTITY
TYPING
OKE TASK 1OKE TASK 2RT2KBREPaper
ACE2004Link
AIDA/CoNLL-CompleteLink
AIDA/CoNLL-Test ALink
AIDA/CoNLL-Test BLink
AIDA/CoNLL-TrainingLink
AQUAINT-
CoNLL2003Link
DBpediaSpotlightLink
DerczynskiLink
ERD2014Link
GERDAQ-DevLink
GERDAQ-TestLink
GERDAQ-TrainingALink
GERDAQ-TrainingBLink
IITBLink
KORE50Link
MSNBCLink
Microposts2013-TestLink
Microposts2013-TrainLink
Microposts2014-TestLink
Microposts2014-TrainLink
Microposts2015-TestLink
Microposts2015-TrainLink
Microposts2016-TestLink
Microposts2016-TrainLink
N3-RSS-500Link
N3-Reuters-128Link
OKE 2015 Task 1Link
OKE 2015 Task 2Link
OKE 2016 Task 1Link
OKE 2016 Task 2Link
OKE 2017 Task 1Link
OKE 2017 Task 2Link
OKE 2017 Task 3Link
OKE 2018 Task 1Link
OKE 2018 Task 2Link
OKE 2018 Task 3Link
OKE 2018 Task 4Link
RitterLink
Senseval 2Link
Senseval 3Link
UMBC-TestLink
UMBC-TrainLink
WSDM 2012Link

Long term stability

The idea of GERBIL emerged in September 2014 when a couple of articles released at the same time claimed to be state-of-the-art. Especially, those approaches were not easily comparable due to their heterogeneous set-up, dataset use and evaluation metrics. Thus, we decided to build GERBIL and extend the BAT-Framework to break the barriers for people not able to write source code.

GERBIL is now more than 3 years old and has hosted more than 50.000 experiments. It is currently hosted at the research and development unit of the University Leipzig Computation Center and the Paderborn University which keep daily backups to ensure long-term quotability.

The survey data from our paper can be found at GERBIL's GitHub repository.

Contributors

In alphabetic order.

  • Ciro Baron (University Leipzig, Germany)
  • Andreas Both (R&D, Unister GmbH, Germany)
  • Martin Brümmer (University Leipzig, Germany)
  • Diego Ceccarelli (Unversity Pisa, Italy)
  • Marco Cornolti (University of Pisa, Italy)
  • Didier Cherix (R&D, Unister GmbH, Germany)
  • Bernd Eickmann (R&D, Unister GmbH, Germany)
  • Paolo Ferragina (University of Pisa, Italy)
  • Christiane Lemke (R&D, Unister GmbH, Germany)
  • Andrea Moro (Sapienza University of Rome, Italy)
  • Roberto Navigli (Sapienza University of Rome, Italy)
  • Francesco Piccinno (University of Pisa, Italy)
  • Giuseppe Rizzo (EURECOM, France)
  • Harald Sack (HPI Potsdam, Germany)
  • René Speck (Institute for Applied Informatics, Germany)
  • Raphaël Troncy (EURECOM, France)
  • Jörg Waitelonis (HPI Potsdam, Germany)
  • Lars Wesemann (R&D, Unister GmbH, Germany)
Maintainer
Michael Röder

Publications

No papers found