GERBIL
Production-ready project (September 2014 - Ongoing)
General Entity Annotation Benchmark Framework
About the project
- If you want to know more, please have a look at our novel paper about GERBIL at the Semantic Web Journal
- We also used GERBIL for benchmarking system for question answering
Available Annotators
| BAT-FRAMEWORK | GERBIL 1.0.0 | GERBIL1.2.5 | Experiment | Paper | ||
|---|---|---|---|---|---|---|
| AGDISTIS | (✔) | ✔ | ✔ | D2KB | | |
| AIDA | ✔ | ✔ | ✔ | A2KB | | |
| Babely | ✔ | ✔ | A2KB | | ||
| CETUS | ✔ | OKE Task 2 | | |||
| CETUS (FOX) | ✔ | OKE Task 2 | | |||
| DBpedia Spotlight | ✔ | ✔ | ✔ | A2KB | | |
| Dexter | ✔ | ✔ | A2KB | | ||
| DoSeR (*) | ✔ | D2KB | | |||
| entityclassifier.eu NER | ✔ | A2KB | | |||
| FOX | ✔ | OKE Task 1 | | |||
| FRED | ✔ | OKE Task 1 | | |||
| FREME NER | ✔ | OKE Task 1 | | |||
| KEA | ✔ | ✔ | A2KB | | ||
| NERD-ML | ✔ | ✔ | A2KB | | ||
| NERFGUN | ✔ | D2KB | | |||
| OpenTapioca | ✔ | A2KB | | |||
| PBOH | ✔ | D2KB | | |||
| TagMe 2 | ✔ | ✔ | ✔ | A2KB | | |
| WAT | ✔ | ✔ | A2KB | | ||
| xLisa | ✔ | A2KB | |
(*) Annotator isn't available any more
Supported Experiments for each Annotator
The following table lists the annotators that are currently available and the experiment types they support. Note that some of the A2KB annotators support the D2KB experiment by offering an own API method. Other A2KB annotators can be chosen for a D2KB experiment as well as described in the wiki. However, since the comparison might not be fair, we marked these annotators with (✔) in the table. The same is done for Entity Typing.
| A2KB, C2KB, Entity Recognition | D2KB | Entity Typing | OKE TASK 1 | OKE TASK 2 | RT2KB | RE | |
|---|---|---|---|---|---|---|---|
| AGDISTIS | ✔ | ||||||
| AIDA | ✔ | (✔) | |||||
| Babely | ✔ | ✔ | |||||
| CETUS | ✔ | ||||||
| CETUS (FOX) | ✔ | ||||||
| DBpedia Spotlight | ✔ | ✔ | ✔ | ✔ | ✔ | ||
| Dexter | ✔ | (✔) | |||||
| DoSeR (*) | ✔ | ||||||
| entityclassifier.eu NER | ✔ | (✔) | |||||
| FOX | ✔ | (✔) | (✔) | ✔ | ✔ | ✔ | |
| FRED | ✔ | (✔) | (✔) | ✔ | ✔ | ||
| FREME NER | ✔ | ✔ | ✔ | ✔ | ✔ | ||
| KEA | ✔ | ✔ | |||||
| NERD-ML | ✔ | (✔) | |||||
| NERFGUN | ✔ | ||||||
| OpenTapioca | ✔ | ✔ | |||||
| PBOH | ✔ | ||||||
| TagMe 2 | ✔ | (✔) | |||||
| WAT | ✔ | ✔ | |||||
| xLisa | ✔ | (✔) |
(*) Annotator isn't available any more
Available Datasets
The following table lists the datasets that are currently available and the experiment types they support.
| A2KB, C2KB, D2KB, Entity Recognition | Entity Typing | OKE TASK 1 | OKE TASK 2 | RT2KB | RE | Paper | |
|---|---|---|---|---|---|---|---|
| ACE2004 | ✔ | ||||||
| AIDA/CoNLL-Complete | ✔ | ||||||
| AIDA/CoNLL-Test A | ✔ | ||||||
| AIDA/CoNLL-Test B | ✔ | ||||||
| AIDA/CoNLL-Training | ✔ | ||||||
| AQUAINT | ✔ | - | |||||
| CoNLL2003 | |||||||
| DBpediaSpotlight | ✔ | ||||||
| Derczynski | ✔ | ||||||
| ERD2014 | ✔ | ||||||
| GERDAQ-Dev | ✔ | ||||||
| GERDAQ-Test | ✔ | ||||||
| GERDAQ-TrainingA | ✔ | ||||||
| GERDAQ-TrainingB | ✔ | ||||||
| IITB | ✔ | ||||||
| KORE50 | ✔ | ||||||
| MSNBC | ✔ | ||||||
| Microposts2013-Test | ✔ | ✔ | |||||
| Microposts2013-Train | ✔ | ✔ | |||||
| Microposts2014-Test | ✔ | ||||||
| Microposts2014-Train | ✔ | ||||||
| Microposts2015-Test | ✔ | ||||||
| Microposts2015-Train | ✔ | ||||||
| Microposts2016-Test | ✔ | ||||||
| Microposts2016-Train | ✔ | ||||||
| N3-RSS-500 | ✔ | ||||||
| N3-Reuters-128 | ✔ | ||||||
| OKE 2015 Task 1 | ✔ | ✔ | ✔ | ✔ | |||
| OKE 2015 Task 2 | ✔ | ||||||
| OKE 2016 Task 1 | ✔ | ✔ | ✔ | ✔ | |||
| OKE 2016 Task 2 | ✔ | ||||||
| OKE 2017 Task 1 | ✔ | ||||||
| OKE 2017 Task 2 | ✔ | ||||||
| OKE 2017 Task 3 | ✔ | ✔ | ✔ | ✔ | |||
| OKE 2018 Task 1 | ✔ | ||||||
| OKE 2018 Task 2 | ✔ | ||||||
| OKE 2018 Task 3 | ✔ | ||||||
| OKE 2018 Task 4 | ✔ | ✔ | |||||
| Ritter | ✔ | ✔ | ✔ | ||||
| Senseval 2 | ✔ | ||||||
| Senseval 3 | ✔ | ||||||
| UMBC-Test | ✔ | ✔ | ✔ | ||||
| UMBC-Train | ✔ | ✔ | ✔ | ||||
| WSDM 2012 | ✔ |
Long term stability
The idea of GERBIL emerged in September 2014 when a couple of articles released at the same time claimed to be state-of-the-art. Especially, those approaches were not easily comparable due to their heterogeneous set-up, dataset use and evaluation metrics. Thus, we decided to build GERBIL and extend the BAT-Framework to break the barriers for people not able to write source code.
GERBIL is now more than 3 years old and has hosted more than 50.000 experiments. It is currently hosted at the research and development unit of the University Leipzig Computation Center and the Paderborn University which keep daily backups to ensure long-term quotability.
The survey data from our paper can be found at GERBIL's GitHub repository.
Contributors
The main developer of the project is Michael Röder.
We thank Ricardo Usbeck for the initial creation of the project and the development of the main idea. We also thank Lixi Conrads for the large amount of development that they invested into the project.
Other people who contributed to the project are (in alphabetic order):
- Ciro Baron (University Leipzig, Germany)
- Lukas Blübaum (DICE group, Germany)
- Andreas Both (R&D, Unister GmbH, Germany)
- Martin Brümmer (University Leipzig, Germany)
- Diego Ceccarelli (Unversity Pisa, Italy)
- Marco Cornolti (University of Pisa, Italy)
- Didier Cherix (R&D, Unister GmbH, Germany)
- Bernd Eickmann (R&D, Unister GmbH, Germany)
- Paolo Ferragina (University of Pisa, Italy)
- Christiane Lemke (R&D, Unister GmbH, Germany)
- Andrea Moro (Sapienza University of Rome, Italy)
- Roberto Navigli (Sapienza University of Rome, Italy)
- Francesco Piccinno (University of Pisa, Italy)
- Giuseppe Rizzo (EURECOM, France)
- Harald Sack (HPI Potsdam, Germany)
- René Speck (DICE group, Germany)
- Nikit Srivastava (DICE group, Germany)
- Raphaël Troncy (EURECOM, France)
- Jörg Waitelonis (HPI Potsdam, Germany)
- Lars Wesemann (R&D, Unister GmbH, Germany)
We also thank all the contributers on Github.
