An extension of the GERBIL benchmarking platform

Project Group Master

An extension of the GERBIL benchmarking platform

GERBIL is a benchmarking platform. Originally developed for benchmarking knowledge extraction systems, it emerged into a platform that is used to benchmark question answering, fact validation and machine translation systems as well. However, this let to a situation in which branches started to diverge and updates have to be applied to one branch after the other. Hence, the first goal of this PG is to merge all these GERBIL versions into one single GERBIL that comprises a task independent core and task dependent modules. The second goal is to use this core to implement an extension of GERBIL for benchmarking knowledge graph embedding algorithms.

Create a generic GERBIL framework that is not bound to a particular set of tasks.
Use the previously created generic framework to develop a new GEBRIL instance for knowledge graph embeddings

Literature

"GERBIL – General Entity Annotation Benchmark Framework" By Ricardo Usbeck, Michael Röder, Axel-Cyrille Ngonga Ngomo, Ciro Baron, Andreas Both, Martin Brümmer, Diego Ceccarelli, Marco Cornolti, Didier Cherix, Bernd Eickmann, Paolo Ferragina, Christiane Lemke, Andrea Moro, Roberto Navigli, Francesco Piccinno, Giuseppe Rizzo, Harald Sack, René Speck, Raphaël Troncy, Jörg Waitelonis, Lars Wesemann. In Proceedings of the 24th WWW conference, 2015.
"GERBIL - Benchmarking Named Entity Recognition and Linking consistently" By Michael Röder, Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo. In Semantic Web Journal, 2018.
"Benchmarking Question Answering Systems" By Ricardo Usbeck, Michael Röder, Michael Hoffmann, Felix Conrads, Jonathan Huthmann, Axel-Cyrille Ngonga Ngomo, Christian Demmler, Christina Unger. In Semantic Web Journal, 2019.
"A general benchmarking framework for text generation" By Diego Moussallem, Paramjot Kaur, Thiago Ferreira, Chris van~der Lee, Anastasia Shimorina, Felix Conrads, Michael Röder, René Speck, Claire Gardent, Simon Mille, Nikolai Ilinykh, and Axel-Cyrille Ngonga~Ngomo. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), 2020.

Project presentation

The slides of the presentation can be found here.

Question & Answer Session

Q: Will there be prior assignments? What would these assignments look like?
A: Yes, there will be assignments to ensure that participants are able to program in Java. To this end, the assignments will be simple Java programming tasks.

Q: Is there a seminar connected to this PG?
A: No.

Q: What are the prerequisites for this PG?
A: You should be able to program in Java. Knowledge about RDF, knowledge graphs, embedding algorithms and software design principles is beneficial.

In case you have further questions, feel free to contact Michael Röder.