automatically generating problems and solutions for ... · pdf fileumair@iitk.ac.in sumit...

# Premises = 2, # Vars = 3, # Inference steps = 2,

Inference Rules = {Modus Tollens, Simplification}

Automatically Generating Problems and Solutionsfor Natural Deduction

Natural Deduction• Method for establishing the validity of propositional

type arguments, where the conclusion is derived

from the premises through a series of discrete steps,

by application of some inference/replacement rule.

• Typically taught as part of an introductory course on

logic, which is a central component of college

education.

GoalBuilding an efficient computer-aided education system

for natural deduction that can perform

1. Solution Generation: step-by-step proof generation of

problems.

2. Problem Generation: Generate problems

a) having similar solution to a seed problem.

b) satisfying specified parameters.

Key Insights• Small-sized hypothesis: Propositions

occurring in educational contexts use small

number of variables and have small size.

• Truth-Table Representation: A proposition

can be abstracted using its truth-table,

which can be represented using a bitvector

representation.

• Offline Computation: The symbolic reasoning

required to pattern match propositions for

applying inference rules can be performed

and stored in an offline phase.

Contributions• We propose leveraging the key insights for building efficient

computer-aided education system for natural deduction.

• A novel two-phased methodology for solution generation that

first searches for an abstract solution and then refines it to a

concrete solution.

• A novel methodology for generating problems using a

process that is reverse of solution generation.

• We present detailed experimental results on 279 benchmark

textbook problems. Our tool can solve 84% of these and is

able to generate few thousands of similar / parameterized

problems on average per instance in a few minutes.

Conclusion• Computer-aided instruction can raise the quality of education by making it more interactive & customized.

• Our tool can free instructors from the burden of generating similar difficulty level variants for assignment problems

and creating their sample solutions, which could be useful in both Massive Open Online Courses (MOOCs) and

traditional classroom settings.

Universal Proof GraphThe key data structure used is an offline computed hyper-graph whose

nodes are truth table bit-vectors, and hyper-edges are inference rules.

Simp.

M.P. , M.T.

M.P. , M.T.

Conj.

Add.Add.

204240

15241

51

0243

P Q R ¬P ¬Q P ∧ Q P → Q P → (Q ∧ R)

0 0 0 1 1 0 1 1

0 0 1 1 1 0 1 1

0 1 0 1 0 0 1 1

0 1 1 1 0 0 1 1

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0

1 1 0 0 0 1 1 0

1 1 1 0 0 1 1 1

15 51 85 240 204 3 243 241

Truth Table

Umair Z. Ahmedumair@iitk.ac.in

Sumit Gulwanisumitg@microsoft.com

Amey Karkare karkare@cse.iitk.ac.in

Premise-1 Premise-2 Conclusion

P → (Q ∧ R) ¬Q ¬P

Original Problem

Abstract ProofStep Truth-Table Reason

P1 241 Premise

P2 204 Premise

1 243 P1, Simplification

2 240 1, P2, Modus Tollens

Natural Deduction ProofStep Truth-Table Proposition Reason

P1 241 P → (Q ∧ R) Premise

P2 204 ¬Q Premise

1 241 ¬P ∨ (Q ∧ R) P1, Implication

2 241 (¬P ∨ Q) ∧ (¬P ∨ R) 1, Distribution

3 243 ¬P ∨ Q 2, Simplification

4 243 P → Q 3, Implication

5 240 ¬P 4, P2, Modus Tollens

3Conj.

Premise-1 Premise-2 Conclusion

Q ≡ ¬P P ∧ (Q ∨ R) P ∧ R

P ∧ Q Q → (P ∧ R) P ∧ R

(P → R) → Q (R → Q) → P P

Similar Problem Generation

Parameterized Problem Generation

Premise-1 Premise-2 Conclusion

P ∧ (R ∨ Q) P → (¬Q ∧ R) P ∧ R

R ≡ P (R ∨ P) ∧ (R ≡ ¬ Q) P ∧ R

(Q ≡ P) → (P ∧ R) (R ≡ P) ∧ (Q ≡ P) P ∧ R