"cmake/vscode:/vscode.git/clone" did not exist on "74d88bf832a52b7f37b0b9383a9fdb0b91eda346"
Commit 0375c800 authored by Jay Young's avatar Jay Young Committed by GitHub
Browse files

Fix `Segmentation Fault` when loading very big corpus file

example_pos_ is the cursor to current position when reading the corpus file. When corpus file is very big, the corpus_size_ may bigger than max of int32, then example_pos_ became negative, and cause `corpus_[example_pos_]` to `Segmentation Fault`.
parent 3bf85a4e
...@@ -128,7 +128,7 @@ class SkipgramWord2vecOp : public OpKernel { ...@@ -128,7 +128,7 @@ class SkipgramWord2vecOp : public OpKernel {
random::SimplePhilox rng_ GUARDED_BY(mu_); random::SimplePhilox rng_ GUARDED_BY(mu_);
int32 current_epoch_ GUARDED_BY(mu_) = -1; int32 current_epoch_ GUARDED_BY(mu_) = -1;
int64 total_words_processed_ GUARDED_BY(mu_) = 0; int64 total_words_processed_ GUARDED_BY(mu_) = 0;
int32 example_pos_ GUARDED_BY(mu_); int64 example_pos_ GUARDED_BY(mu_);
int32 label_pos_ GUARDED_BY(mu_); int32 label_pos_ GUARDED_BY(mu_);
int32 label_limit_ GUARDED_BY(mu_); int32 label_limit_ GUARDED_BY(mu_);
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment