Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
ResNet50_tensorflow
Commits
5c3e08b7
Commit
5c3e08b7
authored
Mar 23, 2021
by
A. Unique TensorFlower
Browse files
Merge pull request #9670 from ChristopherBrix:nlp_data_tagging_quadratic
PiperOrigin-RevId: 364726354
parents
23ef9155
723d96c5
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
3 deletions
+3
-3
official/nlp/data/tagging_data_lib.py
official/nlp/data/tagging_data_lib.py
+3
-3
No files found.
official/nlp/data/tagging_data_lib.py
View file @
5c3e08b7
...
...
@@ -224,10 +224,10 @@ def _tokenize_example(example, max_length, tokenizer, text_preprocessing=None):
max_length
=
max_length
-
2
new_examples
=
[]
new_example
=
InputExample
(
sentence_id
=
example
.
sentence_id
,
sub_sentence_id
=
0
)
for
i
,
word
in
enumerate
(
example
.
words
):
if
any
([
x
<
0
for
x
in
example
.
label_ids
]):
raise
ValueError
(
"Unexpected negative label_id: %s"
%
example
.
label_ids
)
if
any
([
x
<
0
for
x
in
example
.
label_ids
]):
raise
ValueError
(
"Unexpected negative label_id: %s"
%
example
.
label_ids
)
for
i
,
word
in
enumerate
(
example
.
words
):
if
text_preprocessing
:
word
=
text_preprocessing
(
word
)
subwords
=
tokenizer
.
tokenize
(
word
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment