Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
75ff5305
"tests/models/vscode:/vscode.git/clone" did not exist on "12c39e5693f7223be162a1e84de026a6545029eb"
Unverified
Commit
75ff5305
authored
Jan 04, 2021
by
Patrick von Platen
Committed by
GitHub
Jan 04, 2021
Browse files
correct docs (#9378)
parent
ec54d70e
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
6 additions
and
3 deletions
+6
-3
docs/source/custom_datasets.rst
docs/source/custom_datasets.rst
+6
-3
No files found.
docs/source/custom_datasets.rst
View file @
75ff5305
...
...
@@ -558,12 +558,15 @@ we can use the built in :func:`~transformers.BatchEncoding.char_to_token` method
end_positions
=
[]
for
i
in
range
(
len
(
answers
)):
start_positions
.
append
(
encodings
.
char_to_token
(
i
,
answers
[
i
][
'answer_start'
]))
end_positions
.
append
(
encodings
.
char_to_token
(
i
,
answers
[
i
][
'answer_end'
]
-
1
))
#
if
None
,
the
answer
passage
has
been
truncated
end_positions
.
append
(
encodings
.
char_to_token
(
i
,
answers
[
i
][
'answer_end'
]))
#
if
start
position
is
None
,
the
answer
passage
has
been
truncated
if
start_positions
[-
1
]
is
None
:
start_positions
[-
1
]
=
tokenizer
.
model_max_length
#
if
end
position
is
None
,
the
'char_to_token'
function
points
to
the
space
before
the
correct
token
-
>
add
+
1
if
end_positions
[-
1
]
is
None
:
end_positions
[-
1
]
=
tokenizer
.
model_max_length
end_positions
[-
1
]
=
encodings
.
char_to_token
(
i
,
answers
[
i
][
'answer_end'
]
+
1
)
encodings
.
update
({
'start_positions'
:
start_positions
,
'end_positions'
:
end_positions
})
add_token_positions
(
train_encodings
,
train_answers
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment