PUB - Publikationen an der Universität Bielefeld

The ability to reason about events and their temporal relations is a key aspect in Natural Language Understanding. In this paper, we investigate the ability of Large Language Models to resolve temporal references with respect to longer event sets. Given that events rarely occur in isolation, it is crucial to determine the extent to which Large Language Models can reason about longer sets of events. Towards this goal, we introduce a novel synthetic benchmark dataset comprising of 2,200 questions to test the abilities of LLMs to reason about events using a Question Answering task as proxy. We compare the performance of 4 state of the art LLMs on the benchmark, analyzing their performance in dependence of the length of the event set considered as well as of the explicitness of the temporal reference. Our results show that, while the benchmarked LLMs can answer questions over event sets with a handful of events and explicit temporal references successfully, performance clearly deteriorat es with larger event set length and when temporal references get less explicit. The Benchmark is available at [https://gitlab.ub.uni-bielefeld.de/s.kenneweg/bamer](https://gitlab.ub.uni-bielefeld.de/s.kenneweg/bamer)