增加transformer-xl模型代码

TensorFlow/NLP/transformer-xl-master/.idea/.gitignore

+# Default ignored files
+/shelf/
+/workspace.xml

TensorFlow/NLP/transformer-xl-master/.idea/inspectionProfiles/profiles_settings.xml

+<component name="InspectionProjectProfileManager">
+  <settings>
+    <option name="USE_PROJECT_PROFILE" value="false" />
+    <version value="1.0" />
+  </settings>
+</component>
\ No newline at end of file

TensorFlow/NLP/transformer-xl-master/.idea/misc.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="ProjectRootManager" version="2" project-jdk-name="Python 3.8" project-jdk-type="Python SDK" />
+</project>
\ No newline at end of file

TensorFlow/NLP/transformer-xl-master/.idea/modules.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="ProjectModuleManager">
+    <modules>
+      <module fileurl="file://$PROJECT_DIR$/.idea/transformer-xl-master.iml" filepath="$PROJECT_DIR$/.idea/transformer-xl-master.iml" />
+    </modules>
+  </component>
+</project>
\ No newline at end of file

TensorFlow/NLP/transformer-xl-master/.idea/transformer-xl-master.iml

+<?xml version="1.0" encoding="UTF-8"?>
+<module type="PYTHON_MODULE" version="4">
+  <component name="NewModuleRootManager">
+    <content url="file://$MODULE_DIR$" />
+    <orderEntry type="inheritedJdk" />
+    <orderEntry type="sourceFolder" forTests="false" />
+  </component>
+  <component name="PyDocumentationSettings">
+    <option name="format" value="PLAIN" />
+    <option name="myDocStringFormat" value="Plain" />
+  </component>
+</module>
\ No newline at end of file

TensorFlow/NLP/transformer-xl-master/README.md

+参考网址https://github.com/kimiyoung/transformer-xl
+
+# 1.数据准备
+
+数据集下载地址：http://mattmahoney.net/dc/enwik8.zip
+
+数据处理：在enwik8.zip同一文件夹下
+
+```
+python3 prep_enwik8.py
+```
+
+# 2.数据转换
+
+1.修改enwik8_base_gpu.sh脚本中DATA_ROOT为数据路径
+
+2.执行
+
+```
+bash ./scripts/enwik8_base_gpu.sh train_data
+```
+
+# 3.模型运行
+
+```
+#单卡运行
+bash ./scripts/enwik8_base1_gpu.sh train
+#多卡运行
+bash scripts/enwik8_base_gpu.sh
+```
+
+# 4.重要参数说明
+
+```
+enwik8_base_gpu.sh & enwik8_base1_gpu.sh
+--MODEL_DIR   --模型保存位置
+--NUM_CORE    --为使用GPU数量，
+--BSZ         --batchs_size（注意使用数据转化是使用的BSZ和训练使用的BSZ需要保持一致）
+--TRAIN_STEPS --训练步数
+```
+

TensorFlow/NLP/transformer-xl-master/README.old.md

+
+## Introduction
+
+This directory contains our TF implementation of Transformer-XL. Note that our state-of-the-art results reported in the paper were obtained by training the model on a large-scale TPU cluster, and our gpu codebase currently does not support distributed training. Here we provide two sets of hyperparameters and scripts:
+- `*large_tpu.sh` are for the SoTA setting on TPUs. These are exactly the commands we used to obtained our best results.
+- `*base_gpu.sh` are for the base models which can be run on a few GPUs.
+
+
+## Prerequisite
+
+- Python 2.7
+- Tensorflow [1.12.0](https://github.com/tensorflow/tensorflow/releases/tag/v1.12.0)
+
+
+
+## Obtain and evaluate pretrained SoTA models
+
+#### 1. Download preprocessed data (vocab) & pretrained models
+
+(a) Set your own `DATA_ROOT` in `sota/download.sh` (default to `./`), which will be the root diretory of downloaded model.
+
+(b) Then, download the model & data by `bash sota/download.sh`. After downloading, the expected directory structure is as follows
+
+```markdown
+pretrained_xl
+  tf_enwik8/
+    data/
+      cache.pkl
+      corpus-info.json
+    model/
+      checkpoint
+      model.ckpt*
+  tf_wt103/
+  	...
+  ...
+```
+
+**Note**: we include preprocessed data in the download files to make sure the **same vocabulary** is used. Please see the code `tf/data_utils.py` to understand the data structure.
+
+
+
+#### 2. Run evaluation scripts to replicate SoTA results on GPUs
+
+- **enwik8**: modify the script `sota/enwik8.sh` accordingly (see below)
+  - set `DATA_ROOT` to the same folder used in the download step (default to `./`)
+  - set `TEST_NUM_CORE ` (number of GPUs to use): we recommend 2 GPUs => about 60 mins
+  - run the script: `bash sota/enwik8.sh`
+
+- **lm1b**: modify the script `sota/lm1b.sh` accordingly  (see below)
+  - set `DATA_ROOT` to the same folder used in the download step (default to `./`)
+  - set `TEST_NUM_CORE ` (number of GPUs to use): we recommend 1 GPUs => less than 5 mins
+  - run the script: `bash sota/lm1b.sh`
+
+- **wt103**:  modify the script `sota/wt103.sh` accordingly  (see below)
+  - set `DATA_ROOT` to the same folder used in the download step (default to `./`)
+  - set `TEST_NUM_CORE ` (number of GPUs to use): we recommend 1 GPUs => less than 5 mins
+  - run the script: `bash sota/wt103.sh`
+
+- **text8**:  modify the script `sota/text8.sh` accordingly  (see below)
+  - set `DATA_ROOT` to the same folder used in the download step (default to `./`)
+  - set `TEST_NUM_CORE ` (number of GPUs to use): we recommend 2 GPUs => about 60 mins
+  - run the script: `bash sota/text8.sh`
+
+
+#### 3. Resources Needed for SoTA Model Training
+
+We used 32, 32, 64, and 512 TPU cores for training our best models on enwik8, text8, wt103, and lm1b respectively. The training time for each model ranges from 2 to 5 days.
+
+
+
+## Train "Transformer-XL" from scratch with GPUs or TPUs
+
+### 1. Download raw data
+
+`bash getdata.sh`
+
+
+
+### 2. Preprocess, training and evaluation
+
+For `dataset` in `[enwik8, lm1b, wt103, text8]`:
+
+- check out `scripts/dataset_base_gpu.sh` for GPU training and evaluation
+- check out `scripts/dataset_large_tpu.sh` for TPU training and evaluation
+
+
+
+#### (1) Preprocess raw data and create tfrecords
+
+**NOTE**: The preprocessing for GPU and TPU are different. So, you have to run them separately.
+
+GPU:
+
+- create training and validation data: `bash scripts/dataset_bas_gpu.sh train_data`
+- create test data: `bash scripts/dataset_base_gpu.sh test_data`
+
+TPU:
+
+- Set the Google storage URL  in `scripts/dataset_large_tpu.sh`:
+  - `GSDATA`: data URL
+  - `GSEXP`: experiment URL
+- create training and validation data: `bash scripts/dataset_large_tpu.sh train_data`
+- create test data: `bash scripts/dataset_large_tpu.sh test_data`
+
+
+
+#### (2) Run training
+
+Base models on GPUs:
+
+- Modify the configurations in `scripts/dataset_base_gpu.sh`  according to your needs.
+- `bash scripts/dataset_base_gpu.sh train`
+- If enough resources are available, increasing the model sizes (e.g., `N_LAYER`, `D_MODEL`, `D_EMBED`, `D_HEAD`, `D_INNER`) so that they are closer to the values defined in `scripts/dataset_large_tpu.sh`. Likewise, when resources are limited, decrease the model sizes. It is recommended to ensure that `D_MODEL == D_EMBED` and `D_MODEL == N_HEAD x D_HEAD`. When the model sizes increase, remember to increase `warmup_steps` accordingly to alleviate optimization difficulties.
+- Adjust the `NUM_CORE` parameter to reflect the number of GPUs to use.
+
+Larger models on TPUs:
+
+- Modify the configurations in `scripts/dataset_large_tpu.sh`  according to your needs.
+- `bash scripts/dataset_large_tpu.sh train`
+
+
+
+#### (3) Run evaluation
+
+Base models on GPUs:
+
+- `bash scripts/dataset_base_gpu.sh eval --eval_ckpt_path PATH_TO_CKPT`
+
+Larger models on TPUs:
+
+- `bash scripts/dataset_base_tpu.sh eval --eval_ckpt_path PATH_TO_CKPT`

TensorFlow/NLP/transformer-xl-master/avg_checkpoints.py

+# coding=utf-8
+# Copyright 2018 The Tensor2Tensor Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Script to average values of variables in a list of checkpoint files."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import numpy as np
+import six
+from six.moves import zip  # pylint: disable=redefined-builtin
+import tensorflow as tf
+
+flags = tf.flags
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string("checkpoints", "",
+                    "Comma-separated list of checkpoints to average.")
+flags.DEFINE_integer("num_last_checkpoints", 0,
+                     "Averages the last N saved checkpoints."
+                     " If the checkpoints flag is set, this is ignored.")
+flags.DEFINE_string("prefix", "",
+                    "Prefix (e.g., directory) to append to each checkpoint.")
+flags.DEFINE_string("output_path", "/tmp/averaged.ckpt",
+                    "Path to output the averaged checkpoint to.")
+
+
+def checkpoint_exists(path):
+  return (tf.gfile.Exists(path) or tf.gfile.Exists(path + ".meta") or
+          tf.gfile.Exists(path + ".index"))
+
+
+def main(_):
+  tf.logging.set_verbosity(tf.logging.INFO)
+  if FLAGS.checkpoints:
+    # Get the checkpoints list from flags and run some basic checks.
+    checkpoints = [c.strip() for c in FLAGS.checkpoints.split(",")]
+    checkpoints = [c for c in checkpoints if c]
+    if not checkpoints:
+      raise ValueError("No checkpoints provided for averaging.")
+    if FLAGS.prefix:
+      checkpoints = [FLAGS.prefix + c for c in checkpoints]
+  else:
+    assert FLAGS.num_last_checkpoints >= 1, "Must average at least one model"
+    assert FLAGS.prefix, ("Prefix must be provided when averaging last"
+                          " N checkpoints")
+    checkpoint_state = tf.train.get_checkpoint_state(
+        os.path.dirname(FLAGS.prefix))
+    # Checkpoints are ordered from oldest to newest.
+    checkpoints = checkpoint_state.all_model_checkpoint_paths[
+        -FLAGS.num_last_checkpoints:]
+
+  checkpoints = [c for c in checkpoints if checkpoint_exists(c)]
+  if not checkpoints:
+    if FLAGS.checkpoints:
+      raise ValueError(
+          "None of the provided checkpoints exist. %s" % FLAGS.checkpoints)
+    else:
+      raise ValueError("Could not find checkpoints at %s" %
+                       os.path.dirname(FLAGS.prefix))
+
+  # Read variables from all checkpoints and average them.
+  tf.logging.info("Reading variables and averaging checkpoints:")
+  for c in checkpoints:
+    tf.logging.info("%s ", c)
+  var_list = tf.contrib.framework.list_variables(checkpoints[0])
+  var_values, var_dtypes = {}, {}
+  for (name, shape) in var_list:
+    if not name.startswith("global_step"):
+      var_values[name] = np.zeros(shape)
+  for checkpoint in checkpoints:
+    reader = tf.contrib.framework.load_checkpoint(checkpoint)
+    for name in var_values:
+      tensor = reader.get_tensor(name)
+      var_dtypes[name] = tensor.dtype
+      var_values[name] += tensor
+    tf.logging.info("Read from checkpoint %s", checkpoint)
+  for name in var_values:  # Average.
+    var_values[name] /= len(checkpoints)
+
+  with tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE):
+    tf_vars = [
+        tf.get_variable(v, shape=var_values[v].shape, dtype=var_dtypes[v])
+        for v in var_values
+    ]
+  placeholders = [tf.placeholder(v.dtype, shape=v.shape) for v in tf_vars]
+  assign_ops = [tf.assign(v, p) for (v, p) in zip(tf_vars, placeholders)]
+  global_step = tf.Variable(
+      0, name="global_step", trainable=False, dtype=tf.int64)
+  saver = tf.train.Saver(tf.all_variables())
+
+  # Build a model consisting only of variables, set them to the average values.
+  with tf.Session() as sess:
+    sess.run(tf.initialize_all_variables())
+    for p, assign_op, (name, value) in zip(placeholders, assign_ops,
+                                           six.iteritems(var_values)):
+      sess.run(assign_op, {p: value})
+    # Use the built saver to save the averaged checkpoint.
+    saver.save(sess, FLAGS.output_path, global_step=global_step)
+
+  tf.logging.info("Averaged checkpoints saved in %s", FLAGS.output_path)
+
+
+if __name__ == "__main__":
+  tf.app.run()

TensorFlow/NLP/transformer-xl-master/gpu_utils.py

+import os
+import tensorflow as tf
+
+def assign_to_gpu(gpu=0, ps_dev="/device:CPU:0"):
+    def _assign(op):
+        node_def = op if isinstance(op, tf.NodeDef) else op.node_def
+        if node_def.op == "Variable":
+            return ps_dev
+        else:
+            return "/gpu:%d" % gpu
+    return _assign
+
+
+def average_grads_and_vars(tower_grads_and_vars):
+    def average_dense(grad_and_vars):
+        if len(grad_and_vars) == 1:
+            return grad_and_vars[0][0]
+
+        grad = grad_and_vars[0][0]
+        for g, _ in grad_and_vars[1:]:
+            grad += g
+        return grad / len(grad_and_vars)
+
+    def average_sparse(grad_and_vars):
+        if len(grad_and_vars) == 1:
+            return grad_and_vars[0][0]
+
+        indices = []
+        values = []
+        for g, _ in grad_and_vars:
+            indices += [g.indices]
+            values += [g.values]
+        indices = tf.concat(indices, 0)
+        values = tf.concat(values, 0) / len(grad_and_vars)
+        return tf.IndexedSlices(values, indices, grad_and_vars[0][0].dense_shape)
+
+    average_grads_and_vars = []
+    for grad_and_vars in zip(*tower_grads_and_vars):
+        if grad_and_vars[0][0] is None:
+            grad = None
+        elif isinstance(grad_and_vars[0][0], tf.IndexedSlices):
+            grad = average_sparse(grad_and_vars)
+        else:
+            grad = average_dense(grad_and_vars)
+        # Keep in mind that the Variables are redundant because they are shared
+        # across towers. So .. we will just return the first tower's pointer to
+        # the Variable.
+        v = grad_and_vars[0][1]
+        grad_and_var = (grad, v)
+        average_grads_and_vars.append(grad_and_var)
+    return average_grads_and_vars
+
+
+def load_from_checkpoint(saver, logdir):
+    sess = tf.get_default_session()
+    ckpt = tf.train.get_checkpoint_state(logdir)
+    if ckpt and ckpt.model_checkpoint_path:
+        if os.path.isabs(ckpt.model_checkpoint_path):
+            # Restores from checkpoint with absolute path.
+            saver.restore(sess, ckpt.model_checkpoint_path)
+        else:
+            # Restores from checkpoint with relative path.
+            saver.restore(sess, os.path.join(logdir, ckpt.model_checkpoint_path))
+        return True
+    return False