Commit 2dfcc4f1 authored by Neal Wu's avatar Neal Wu
Browse files

Word2vec can now be run if the users compile the ops on their own

parent a8ab7347
...@@ -9,7 +9,7 @@ tutorials. Brief instructions are below. ...@@ -9,7 +9,7 @@ tutorials. Brief instructions are below.
* [Word2Vec Tutorial](http://tensorflow.org/tutorials/word2vec) * [Word2Vec Tutorial](http://tensorflow.org/tutorials/word2vec)
To download the example text and evaluation data: Assuming you have cloned the git repository, navigate into this directory. To download the example text and evaluation data:
```shell ```shell
wget http://mattmahoney.net/dc/text8.zip -O text8.zip wget http://mattmahoney.net/dc/text8.zip -O text8.zip
...@@ -19,21 +19,17 @@ unzip -p source-archive.zip word2vec/trunk/questions-words.txt > questions-word ...@@ -19,21 +19,17 @@ unzip -p source-archive.zip word2vec/trunk/questions-words.txt > questions-word
rm source-archive.zip rm source-archive.zip
``` ```
Assuming you have cloned the git repository, navigate into this directory and You will need to compile the ops as follows:
run using:
```shell ``` shell
cd models/tutorials/embedding TF_INC=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())')
python word2vec_optimized.py \ g++ -std=c++11 -shared word2vec_ops.cc word2vec_kernels.cc -o word2vec_ops.so -fPIC -I $TF_INC -O2
--train_data=text8 \
--eval_data=questions-words.txt \
--save_path=/tmp/
``` ```
To run the code from sources using bazel: Then run using:
```shell ```shell
bazel run -c opt models/tutorials/embedding/word2vec_optimized -- \ python word2vec_optimized.py \
--train_data=text8 \ --train_data=text8 \
--eval_data=questions-words.txt \ --eval_data=questions-words.txt \
--save_path=/tmp/ --save_path=/tmp/
......
...@@ -17,5 +17,3 @@ ...@@ -17,5 +17,3 @@
from __future__ import absolute_import from __future__ import absolute_import
from __future__ import division from __future__ import division
from __future__ import print_function from __future__ import print_function
import gen_word2vec
...@@ -42,7 +42,7 @@ from six.moves import xrange # pylint: disable=redefined-builtin ...@@ -42,7 +42,7 @@ from six.moves import xrange # pylint: disable=redefined-builtin
import numpy as np import numpy as np
import tensorflow as tf import tensorflow as tf
import gen_word2vec as word2vec word2vec = tf.load_op_library(os.path.dirname(os.path.realpath(__file__)) + '/word2vec_ops.so')
flags = tf.app.flags flags = tf.app.flags
......
...@@ -41,7 +41,7 @@ from six.moves import xrange # pylint: disable=redefined-builtin ...@@ -41,7 +41,7 @@ from six.moves import xrange # pylint: disable=redefined-builtin
import numpy as np import numpy as np
import tensorflow as tf import tensorflow as tf
import gen_word2vec as word2vec word2vec = tf.load_op_library(os.path.dirname(os.path.realpath(__file__)) + '/word2vec_ops.so')
flags = tf.app.flags flags = tf.app.flags
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment