This repository contains the EQUATE dataset, and the Q-REAS symbolic reasoning baseline[1].

EQUATE (Evaluating Quantitative Understanding Aptitude in Textual Entailment) is a new framework for evaluating quantitative reasoning ability in textual entailment.
EQUATE consists of five NLI test sets featuring quantities. You can download EQUATE here. Three of these tests for quantitative reasoning feature language from real-world sources
such as news articles and social media (RTE, NewsNLI Reddit), and two are controlled synthetic tests, evaluating model ability
to reason with quantifiers and perform simple arithmetic (AWP, Stress Test).

Models reporting performance on any NLI dataset can additionally evaluate on the EQUATE benchmark,
to demonstrate competence at quantitative reasoning.

We also provide a baseline quantitative reasoner Q-Reas. Q-Reas manipulates quantity representations symbolically to make entailment decisions.
We hope this provides a framework for the development of hybrid neuro-symbolic architectures to combine the strengths of symbolic reasoners and
neural models.

Q-Reas has five modules:

1. Quantity Segmenter: Extracts quantity mentions
2. Quantity Parser: Parses mentions into semantic representations called NUMSETS
3. Quantity Pruner: Identifies compatible NUMSET pairs
4. ILP Equation Generator: Composes compatible NUMSETS to form plausible equation trees
5. Global Reasoner: Constructs justifications for each quantity in the hypothesis,
   analyzes them to determine entailment labels

You can run Q-Reas on EQUATE with the following command:

python global_reasoner.py -DATASET_NAME (rte, newsnli, reddit, awp, stresstest)

Q-Reas consists of the following components:

1. Quantity Segmenter: quantity_segmenter.py (uses utils_segmenter.py)
2. Quantity Parser: numerical_parser.py (uses utils_parser.py)
3. Quantity Pruner: numset_pruner.py
4. ILP Equation Generator: ilp.py
5. Global Reasoner: global_reasoner.py (uses utils_reasoner.py, scorer.py, eval.py)

and utilizes the following data structures:

1. numset.py: Defines semantic representation for a quantity
2. parsed_numsets.py: Stores extracted NUMSETS for a premise-hypothesis pair
3. compatible_numsets.py: Stores compatible pairs of NUMSETS

Please cite [1] if our work influences your research.

[1] A. Ravichander*, A. Naik*, C. Rose, E. Hovy EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

@article{ravichander2019equate,
  title={EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference},
  author={Ravichander, Abhilasha and Naik, Aakanksha and Rose, Carolyn and Hovy, Eduard},
  journal={arXiv preprint arXiv:1901.03735},
  year={2019}
}