Merge pull request #4484 from thtrieu/master

Added new model research/lm_commonsense

CODEOWNERS

@@ -22,6 +22,7 @@
 /research/lexnet_nc/ @vered1986 @waterson
 /research/lfads/ @jazcollins @susillo
 /research/lm_1b/ @oriolvinyals @panyx0718
+/research/lm_commonsense/ @thtrieu
 /research/marco/ @vincentvanhoucke
 /research/maskgan/ @a-dai
 /research/morph_net/ @gariel-google

research/README.md

@@ -39,6 +39,7 @@ request.
 -   [lfads](lfads): sequential variational autoencoder for analyzing
     neuroscience data.
 -   [lm_1b](lm_1b): language modeling on the one billion word benchmark.
+-   [lm_commonsense](lm_commonsense): commonsense reasoning using language models.
 -   [maskgan](maskgan): text generation with GANs.
 -   [namignizer](namignizer): recognize and generate names.
 -   [neural_gpu](neural_gpu): highly parallel neural computer.

research/lm_commonsense/README.md

+# A Simple Method for Commonsense Reasoning
+
+This repository contains code to reproduce results from [*A Simple Method for Commonsense Reasoning*](TBD).
+
+Authors and contact:
+
+* Trieu H. Trinh (thtrieu@google.com, github: thtrieu)
+* Quoc V. Le (qvl@google.com)
+
+## TL;DR
+
+Commonsense reasoning is a long-standing challenge for deep learning. For example,
+it is difficult to use neural networks to tackle the Winograd Schema dataset - a difficult subset of Pronoun Disambiguation problems. In this work, we use language models to score substitued sentences to decide the correct reference of the ambiguous pronoun (see Figure below for an example). 
+
+![Figure 1. Overview of our method.](method.jpg)
+
+This simple unsupervised method achieves new state-of-the-art (*as of June 1st, 2018*) results on both benchmark PDP-60 and WSC-273 (See Table below), without using rule-based reasoning nor expensive annotated knowledge bases.
+
+| Commonsense-reasoning test  | Previous best result   | Ours  |
+| ----------------------------|:----------------------:|:-----:|
+| Pronoun Disambiguation      | 66.7%                  | 70%   |
+| Winograd Schema Challenge   | 52.8%                  | 63.7% |
+
+
+
+## Citation
+
+If you use our released models below in your publication, please cite the original paper:
+
+@article{TBD}
+
+
+## Requirements
+* Python >=2.6
+* Tensorflow >= v1.4
+* Numpy >= 1.12.1
+
+## Details of this release
+
+The open-sourced components include:
+
+* Test sets from Pronoun Disambiguation Problem (PDP-60) and Winograd Schema Challenges (WSC-273).
+* Tensorflow metagraph and checkpoints of 14 language models (See Appendix A in the paper).
+* A vocabulary file.
+* Code to reproduce results from the original paper.
+
+## How to run
+
+### 1. Download data files
+
+Download all files from the [Google Cloud Storage of this project](https://console.cloud.google.com/storage/browser/commonsense-reasoning/). The easiest way is to install and use `gsutil cp` command-line tool (See [install gsutil](https://cloud.google.com/storage/docs/gsutil_install)).
+
+
+```shell
+# Download everything from the project gs://commonsense-reasoning
+$ gsutil cp -R gs://commonsense-reasoning/* .
+Copying gs://commonsense-reasoning/reproduce/vocab.txt...
+Copying gs://commonsense-reasoning/reproduce/commonsense_test/pdp60.json...     
+Copying gs://commonsense-reasoning/reproduce/commonsense_test/wsc273.json...
+
+...(omitted)
+```
+
+All downloaded content should be in `./reproduce/`. This includes two tests `pdp60.json` and `wsc273.json`, a vocabulary file `vocab.txt` and checkpoints for all 14 language models, each includes three files (`.data`, `.index` and `.meta`). All checkpoint names start with `ckpt-best` since they are saved at the best perplexity on a hold-out text corpus.
+
+```shell
+# Check for the content
+$ ls reproduce/*
+reproduce/vocab.txt
+
+reproduce/commonsense_test:
+pdp60.json  wsc273.json
+
+reproduce/lm01:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm02:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm03:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm04:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm05:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm06:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm07:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm08:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm09:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm10:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm11:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm12:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm13:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+
+reproduce/lm14:
+ckpt-best.data-00000-of-00001  ckpt-best.index  ckpt-best.meta
+```
+
+### 2. Run evaluation code
+
+To reproduce results from the paper, simply run `eval.py` script.
+
+```shell
+$ python eval.py --data_dir=reproduce
+
+Restored from ./reproduce/lm01
+Reset RNN states.
+Processing patch (1, 1) / (2, 4)
+Probs for 
+[['Then' 'Dad' 'figured' ..., 'man' "'s" 'board-bill']
+ ['Then' 'Dad' 'figured' ..., 'man' "'s" 'board-bill']
+ ['Always' 'before' ',' ..., 'now' ',' 'for']
+ ..., 
+ ['Mark' 'was' 'close' ..., 'promising' 'him' ',']
+ ['Mark' 'was' 'close' ..., 'promising' 'him' ',']
+ ['Mark' 'was' 'close' ..., 'promising' 'him' ',']]
+=
+[[  1.64250596e-05   1.77780055e-06   4.14267970e-06 ...,   1.87315454e-03
+    1.57723188e-01   6.31845817e-02]
+ [  1.64250596e-05   1.77780055e-06   4.14267970e-06 ...,   1.87315454e-03
+    1.57723188e-01   6.31845817e-02]
+ [  1.28243030e-07   3.80435935e-03   1.12383246e-01 ...,   9.67682712e-03
+    2.17407525e-01   1.08243264e-01]
+ ..., 
+ [  1.15557734e-04   2.92792241e-03   3.46455898e-04 ...,   2.72328052e-05
+    3.37066874e-02   7.89367408e-02]
+ [  1.15557734e-04   2.92792241e-03   3.46455898e-04 ...,   2.72328052e-05
+    3.37066874e-02   7.89367408e-02]
+ [  1.15557734e-04   2.92792241e-03   3.46455898e-04 ...,   2.72328052e-05
+    3.37066874e-02   7.89367408e-02]]
+Processing patch (1, 2) / (2, 4)
+
+...(omitted)
+
+Accuracy of 1 LM(s) on pdp60 = 0.6
+
+...(omitted)
+
+Accuracy of 5 LM(s) on pdp60 = 0.7
+
+...(omitted)
+
+Accuracy of 10 LM(s) on wsc273 = 0.615
+
+...(omitted)
+
+Accuracy of 14 LM(s) on wsc273 = 0.637
+```
\ No newline at end of file

research/lm_commonsense/eval.py

+# Copyright 2017 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import pickle as pkl
+import numpy as np
+import tensorflow as tf
+import utils
+
+tf.app.flags.DEFINE_string(
+    'data_dir', 'reproduce', 
+    'Path to directory containing data and model checkpoints.')
+
+
+FLAGS = tf.app.flags.FLAGS
+
+
+class EnsembleLM(object):
+  """Ensemble of language models."""
+
+  def __init__(self, test_data_name='wsc273'):
+    vocab_file = os.path.join(FLAGS.data_dir, 'vocab.txt')
+    self.vocab = utils.CharsVocabulary(vocab_file, 50)
+    assert test_data_name in ['pdp60', 'wsc273'], (
+        'Test data must be pdp60 or wsc273, got {}'.format(test_data_name))
+    self.test_data_name = test_data_name
+
+    test_data = utils.parse_commonsense_reasoning_test(test_data_name)
+    self.question_ids, self.sentences, self.labels = test_data
+    self.all_probs = []  # aggregate single-model prediction here.
+
+  def add_single_model(self, model_name='lm1'):
+    """Add a single model into the current ensemble."""
+    # Create single LM
+    single_lm = SingleRecurrentLanguageModel(self.vocab, model_name)
+
+    # Add the single LM prediction.
+    probs = single_lm.assign_probs(self.sentences, self.test_data_name)
+    self.all_probs.append(probs)
+    print('Done adding {}'.format(model_name))
+
+  def evaluate(self):
+    """Evaluate the current ensemble."""
+    # Attach word probabilities and correctness label to each substitution
+    ensembled_probs = sum(self.all_probs) / len(self.all_probs)
+    scorings = []
+    for i, sentence in enumerate(self.sentences):
+      correctness = self.labels[i]
+      word_probs = ensembled_probs[i, :len(sentence)]
+      joint_prob = np.prod(word_probs, dtype=np.float64)
+
+      scorings.append(dict(
+          correctness=correctness,
+          sentence=sentence,
+          joint_prob=joint_prob,
+          word_probs=word_probs))
+    scoring_mode = 'full' if self.test_data_name == 'pdp60' else 'partial'
+    return utils.compare_substitutions(
+        self.question_ids, scorings, scoring_mode)
+
+
+class SingleRecurrentLanguageModel(object):
+  """Single Recurrent Language Model."""
+
+  def __init__(self, vocab, model_name='lm01'):
+    self.vocab = vocab
+    self.log_dir = os.path.join(FLAGS.data_dir, model_name)
+
+  def reset(self):
+    self.sess.run(self.tensors['states_init'])
+
+  def _score(self, word_patch):
+    """Score a matrix of shape (batch_size, num_timesteps+1) str tokens."""
+    word_ids = np.array(
+        [[self.vocab.word_to_id(word) for word in row]
+         for row in word_patch])
+    char_ids = np.array(
+        [[self.vocab.word_to_char_ids(word) for word in row]
+         for row in word_patch])
+    print('Probs for \n{}\n='.format(np.array(word_patch)[:, 1:]))
+
+    input_ids, target_ids = word_ids[:, :-1], word_ids[:, 1:]
+    input_char_ids = char_ids[:, :-1, :]
+
+    softmax = self.sess.run(self.tensors['softmax_out'], feed_dict={
+        self.tensors['inputs_in']: input_ids,
+        self.tensors['char_inputs_in']: input_char_ids
+    })
+
+    batch_size, num_timesteps = self.shape
+    softmax = softmax.reshape((num_timesteps, batch_size, -1))
+    softmax = np.transpose(softmax, [1, 0, 2])
+    probs = np.array([[softmax[row, col, target_ids[row, col]]
+                       for col in range(num_timesteps)]
+                      for row in range(batch_size)])
+    print(probs)
+    return probs
+
+  def _score_patches(self, word_patches):
+    """Score a 2D matrix of word_patches and stitch results together."""
+    batch_size, num_timesteps = self.shape
+    nrow, ncol = len(word_patches), len(word_patches[0])
+    max_len = num_timesteps * ncol
+    probs = np.zeros([0, max_len])  # accumulate results into this.
+
+    # Loop through the 2D matrix of word_patches and score each.
+    for i, row in enumerate(word_patches):
+      print('Reset RNN states.')
+      self.reset()  # reset states before processing each row.
+      row_probs = np.zeros([batch_size, 0])
+      for j, word_patch in enumerate(row):
+        print('Processing patch '
+              '({}, {}) / ({}, {})'.format(i+1, j+1, nrow, ncol))
+        patch_probs = (self._score(word_patch) if word_patch else
+                       np.zeros([batch_size, num_timesteps]))
+        row_probs = np.concatenate([row_probs, patch_probs], 1)
+      probs = np.concatenate([probs, row_probs], 0)
+    return probs
+
+  def assign_probs(self, sentences, test_data_name='wsc273'):
+    """Return prediction accuracy using this LM for a test."""
+
+    probs_cache = os.path.join(self.log_dir, '{}.probs'.format(test_data_name))
+    if os.path.exists(probs_cache):
+      print('Reading cached result from {}'.format(probs_cache))
+      with tf.gfile.Open(probs_cache, 'r') as f:
+        probs = pkl.load(f)
+    else:
+      tf.reset_default_graph()
+      self.sess = tf.Session()
+      # Build the graph.
+      saver = tf.train.import_meta_graph(
+          os.path.join(self.log_dir, 'ckpt-best.meta'))
+      saver.restore(self.sess, os.path.join(self.log_dir, 'ckpt-best'))
+      print('Restored from {}'.format(self.log_dir))
+      graph = tf.get_default_graph()
+      self.tensors = dict(
+          inputs_in=graph.get_tensor_by_name('test_inputs_in:0'),
+          char_inputs_in=graph.get_tensor_by_name('test_char_inputs_in:0'),
+          softmax_out=graph.get_tensor_by_name('SotaRNN_1/softmax_out:0'),
+          states_init=graph.get_operation_by_name('SotaRNN_1/states_init'))
+      self.shape = self.tensors['inputs_in'].shape.as_list()
+
+      # Cut sentences into patches of shape processable by the LM.
+      batch_size, num_timesteps = self.shape
+      word_patches = utils.cut_to_patches(sentences, batch_size, num_timesteps)
+      probs = self._score_patches(word_patches)
+
+      # Cache the probs since they are expensive to evaluate
+      with tf.gfile.Open(probs_cache, 'w') as f:
+        pkl.dump(probs, f)
+    return probs
+
+
+def evaluate_ensemble(test_data_name, number_of_lms):
+  ensemble = EnsembleLM(test_data_name)
+  model_list = ['lm{:02d}'.format(i+1) for i in range(number_of_lms)]
+  for model_name in model_list:
+    ensemble.add_single_model(model_name)
+  accuracy = ensemble.evaluate()
+  print('Accuracy of {} LM(s) on {} = {}'.format(
+      number_of_lms, test_data_name, accuracy))
+
+
+def main(_):
+  evaluate_ensemble('pdp60', 1)  # 60%
+  evaluate_ensemble('pdp60', 5)  # 70%
+  evaluate_ensemble('wsc273', 10)  # 61.5%
+  evaluate_ensemble('wsc273', 14)  # 63.7%
+
+
+if __name__ == '__main__':
+  tf.app.run(main)

research/lm_commonsense/utils.py

+# Copyright 2017 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import json
+import os
+import numpy as np
+import tensorflow as tf
+
+FLAGS = tf.flags.FLAGS
+
+
+class Vocabulary(object):
+  """Class that holds a vocabulary for the dataset."""
+
+  def __init__(self, filename):
+
+    self._id_to_word = []
+    self._word_to_id = {}
+    self._unk = -1
+    self._bos = -1
+    self._eos = -1
+
+    with tf.gfile.Open(filename) as f:
+      idx = 0
+      for line in f:
+        word_name = line.strip()
+        if word_name == '<S>':
+          self._bos = idx
+        elif word_name == '</S>':
+          self._eos = idx
+        elif word_name == '<UNK>':
+          self._unk = idx
+        if word_name == '!!!MAXTERMID':
+          continue
+
+        self._id_to_word.append(word_name)
+        self._word_to_id[word_name] = idx
+        idx += 1
+
+  @property
+  def bos(self):
+    return self._bos
+
+  @property
+  def eos(self):
+    return self._eos
+
+  @property
+  def unk(self):
+    return self._unk
+
+  @property
+  def size(self):
+    return len(self._id_to_word)
+
+  def word_to_id(self, word):
+    if word in self._word_to_id:
+      return self._word_to_id[word]
+    else:
+      if word.lower() in self._word_to_id:
+        return self._word_to_id[word.lower()]
+    return self.unk
+
+  def id_to_word(self, cur_id):
+    if cur_id < self.size:
+      return self._id_to_word[int(cur_id)]
+    return '<ERROR_out_of_vocab_id>'
+
+  def decode(self, cur_ids):
+    return ' '.join([self.id_to_word(cur_id) for cur_id in cur_ids])
+
+  def encode(self, sentence):
+    word_ids = [self.word_to_id(cur_word) for cur_word in sentence.split()]
+    return np.array([self.bos] + word_ids + [self.eos], dtype=np.int32)
+
+
+class CharsVocabulary(Vocabulary):
+  """Vocabulary containing character-level information."""
+
+  def __init__(self, filename, max_word_length):
+    super(CharsVocabulary, self).__init__(filename)
+
+    self._max_word_length = max_word_length
+    chars_set = set()
+
+    for word in self._id_to_word:
+      chars_set |= set(word)
+
+    free_ids = []
+    for i in range(256):
+      if chr(i) in chars_set:
+        continue
+      free_ids.append(chr(i))
+
+    if len(free_ids) < 5:
+      raise ValueError('Not enough free char ids: %d' % len(free_ids))
+
+    self.bos_char = free_ids[0]  # <begin sentence>
+    self.eos_char = free_ids[1]  # <end sentence>
+    self.bow_char = free_ids[2]  # <begin word>
+    self.eow_char = free_ids[3]  # <end word>
+    self.pad_char = free_ids[4]  # <padding>
+
+    chars_set |= {self.bos_char, self.eos_char, self.bow_char, self.eow_char,
+                  self.pad_char}
+
+    self._char_set = chars_set
+    num_words = len(self._id_to_word)
+
+    self._word_char_ids = np.zeros([num_words, max_word_length], dtype=np.int32)
+
+    self.bos_chars = self._convert_word_to_char_ids(self.bos_char)
+    self.eos_chars = self._convert_word_to_char_ids(self.eos_char)
+
+    for i, word in enumerate(self._id_to_word):
+      if i == self.bos:
+        self._word_char_ids[i] = self.bos_chars
+      elif i == self.eos:
+        self._word_char_ids[i] = self.eos_chars
+      else:
+        self._word_char_ids[i] = self._convert_word_to_char_ids(word)
+
+  @property
+  def max_word_length(self):
+    return self._max_word_length
+
+  def _convert_word_to_char_ids(self, word):
+    code = np.zeros([self.max_word_length], dtype=np.int32)
+    code[:] = ord(self.pad_char)
+
+    if len(word) > self.max_word_length - 2:
+      word = word[:self.max_word_length-2]
+    cur_word = self.bow_char + word + self.eow_char
+    for j in range(len(cur_word)):
+      code[j] = ord(cur_word[j])
+    return code
+
+  def word_to_char_ids(self, word):
+    if word in self._word_to_id:
+      return self._word_char_ids[self._word_to_id[word]]
+    else:
+      return self._convert_word_to_char_ids(word)
+
+  def encode_chars(self, sentence):
+    chars_ids = [self.word_to_char_ids(cur_word)
+                 for cur_word in sentence.split()]
+    return np.vstack([self.bos_chars] + chars_ids + [self.eos_chars])
+
+
+_SPECIAL_CHAR_MAP = {
+    '\xe2\x80\x98': '\'',
+    '\xe2\x80\x99': '\'',
+    '\xe2\x80\x9c': '"',
+    '\xe2\x80\x9d': '"',
+    '\xe2\x80\x93': '-',
+    '\xe2\x80\x94': '-',
+    '\xe2\x88\x92': '-',
+    '\xce\x84': '\'',
+    '\xc2\xb4': '\'',
+    '`': '\''
+}
+
+_START_SPECIAL_CHARS = ['.', ',', '?', '!', ';', ':', '[', ']', '\'', '+', '/',
+                        '\xc2\xa3', '$', '~', '*', '%', '{', '}', '#', '&', '-',
+                        '"', '(', ')', '='] + list(_SPECIAL_CHAR_MAP.keys())
+_SPECIAL_CHARS = _START_SPECIAL_CHARS + [
+    '\'s', '\'m', '\'t', '\'re', '\'d', '\'ve', '\'ll']
+
+
+def tokenize(sentence):
+  """Tokenize a sentence."""
+  sentence = str(sentence)
+  words = sentence.strip().split()
+  tokenized = []  # return this
+
+  for word in words:
+    if word.lower() in ['mr.', 'ms.']:
+      tokenized.append(word)
+      continue
+
+    # Split special chars at the start of word
+    will_split = True
+    while will_split:
+      will_split = False
+      for char in _START_SPECIAL_CHARS:
+        if word.startswith(char):
+          tokenized.append(char)
+          word = word[len(char):]
+          will_split = True
+
+    # Split special chars at the end of word
+    special_end_tokens = []
+    will_split = True
+    while will_split:
+      will_split = False
+      for char in _SPECIAL_CHARS:
+        if word.endswith(char):
+          special_end_tokens = [char] + special_end_tokens
+          word = word[:-len(char)]
+          will_split = True
+
+    if word:
+      tokenized.append(word)
+    tokenized += special_end_tokens
+
+  # Add necessary end of sentence token.
+  if tokenized[-1] not in ['.', '!', '?']:
+    tokenized += ['.']
+  return tokenized
+
+
+def parse_commonsense_reasoning_test(test_data_name):
+  """Read JSON test data."""
+  with tf.gfile.Open(os.path.join(
+      FLAGS.data_dir, 'commonsense_test',
+      '{}.json'.format(test_data_name)), 'r') as f:
+    data = json.load(f)
+
+  question_ids = [d['question_id'] for d in data]
+  sentences = [tokenize(d['substitution']) for d in data]
+  labels = [d['correctness'] for d in data]
+
+  return question_ids, sentences, labels
+
+
+PAD = '<padding>'
+
+
+def cut_to_patches(sentences, batch_size, num_timesteps):
+  """Cut sentences into patches of shape (batch_size, num_timesteps).
+
+  Args:
+    sentences: a list of sentences, each sentence is a list of str token.
+    batch_size: batch size
+    num_timesteps: number of backprop step
+
+  Returns:
+    patches: A 2D matrix,
+      each entry is a matrix of shape (batch_size, num_timesteps).
+  """
+  preprocessed = [['<S>']+sentence+['</S>'] for sentence in sentences]
+  max_len = max([len(sent) for sent in preprocessed])
+
+  # Pad to shape [height, width]
+  # where height is a multiple of batch_size
+  # and width is a multiple of num_timesteps
+  nrow = int(np.ceil(len(preprocessed) * 1.0 / batch_size))
+  ncol = int(np.ceil(max_len * 1.0 / num_timesteps))
+  height, width = nrow * batch_size, ncol * num_timesteps + 1
+  preprocessed = [sent + [PAD] * (width - len(sent)) for sent in preprocessed]
+  preprocessed += [[PAD] * width] * (height - len(preprocessed))
+
+  # Cut preprocessed into patches of shape [batch_size, num_timesteps]
+  patches = []
+  for row in range(nrow):
+    patches.append([])
+    for col in range(ncol):
+      patch = [sent[col * num_timesteps:
+                    (col+1) * num_timesteps + 1]
+               for sent in preprocessed[row * batch_size:
+                                        (row+1) * batch_size]]
+      if np.all(np.array(patch)[:, 1:] == PAD):
+        patch = None  # no need to process this patch.
+      patches[-1].append(patch)
+  return patches
+
+
+def _substitution_mask(sent1, sent2):
+  """Binary mask identifying substituted part in two sentences.
+
+  Example sentence and their mask:
+    First sentence  = "I like the cat        's color"
+                       0 0    0   1           0 0
+    Second sentence = "I like the yellow dog 's color"
+                       0 0    0   1      1    0 0
+
+  Args:
+    sent1: first sentence
+    sent2: second sentence
+
+  Returns:
+    mask1: mask for first sentence
+    mask2: mask for second sentence
+  """
+  mask1_start, mask2_start = [], []
+  while sent1[0] == sent2[0]:
+    sent1 = sent1[1:]
+    sent2 = sent2[1:]
+    mask1_start.append(0.)
+    mask2_start.append(0.)
+
+  mask1_end, mask2_end = [], []
+  while sent1[-1] == sent2[-1]:
+    if (len(sent1) == 1) or (len(sent2) == 1):
+      break
+    sent1 = sent1[:-1]
+    sent2 = sent2[:-1]
+    mask1_end = [0.] + mask1_end
+    mask2_end = [0.] + mask2_end
+
+  assert sent1 or sent2, 'Two sentences are identical.'
+  return (mask1_start + [1.] * len(sent1) + mask1_end,
+          mask2_start + [1.] * len(sent2) + mask2_end)
+
+
+def _convert_to_partial(scoring1, scoring2):
+  """Convert full scoring into partial scoring."""
+  mask1, mask2 = _substitution_mask(
+      scoring1['sentence'], scoring2['sentence'])
+
+  def _partial_score(scoring, mask):
+    word_probs = [max(_) for _ in zip(scoring['word_probs'], mask)]
+    scoring.update(word_probs=word_probs,
+                   joint_prob=np.prod(word_probs))
+
+  _partial_score(scoring1, mask1)
+  _partial_score(scoring2, mask2)
+
+
+def compare_substitutions(question_ids, scorings, mode='full'):
+  """Return accuracy by comparing two consecutive scorings."""
+  prediction_correctness = []
+  # Compare two consecutive substitutions
+  for i in range(len(scorings) // 2):
+    scoring1, scoring2 = scorings[2*i: 2*i+2]
+    if mode == 'partial':  # fix joint prob into partial prob
+      _convert_to_partial(scoring1, scoring2)
+
+    prediction_correctness.append(
+        (scoring2['joint_prob'] > scoring1['joint_prob']) ==
+         scoring2['correctness'])
+
+  # Two consecutive substitutions always belong to the same question
+  question_ids = [qid for i, qid in enumerate(question_ids) if i % 2 == 0]
+  assert len(question_ids) == len(prediction_correctness)
+  num_questions = len(set(question_ids))
+
+  # Question is correctly answered only if
+  # all predictions of the same question_id is correct
+  num_correct_answer = 0
+  previous_qid = None
+  correctly_answered = False
+  for predict, qid in zip(prediction_correctness, question_ids):
+    if qid != previous_qid:
+      previous_qid = qid
+      num_correct_answer += int(correctly_answered)
+      correctly_answered = True
+    correctly_answered = correctly_answered and predict
+  num_correct_answer += int(correctly_answered)
+
+  return num_correct_answer / num_questions