ROPES is a reading comprehension dataset that tests a system's ability to apply knowledge from reading to novel situations. In this crowdsourced, 14k question-answering benchmark, a system is required to read a passage of expository text (eg. Wikipedia, textbooks), and use the causal relationships in the text to answer questions about a novel situation.

  • Paper, describing the dataset and our baseline models for it.
  • Data, with over 10K questions in the train set and over 1.6K questions in the dev set (and a similar number in a hidden test set). The data is distributed under the CC BY-SA 4.0 license.
  • Leaderboard with an automated docker-based evaluation on a hidden test set.
  • Citation:

    @inproceedings{Lin2019ReasoningOP,
      author={Kevin Lin and Oyvind Tafjord and Peter Clark and Matt Gardner},
      title={Reasoning Over Paragraph Effects in Situations},
      booktitle={MRQA@EMNLP},
      year={2019}
    }