Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
bfcd5743
Unverified
Commit
bfcd5743
authored
Jun 28, 2022
by
Bill Ray
Committed by
GitHub
Jun 28, 2022
Browse files
In `group_texts` function, drop last block if smaller than `block_size` (#17908)
parent
f71895a6
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
2 additions
and
0 deletions
+2
-0
docs/source/en/tasks/language_modeling.mdx
docs/source/en/tasks/language_modeling.mdx
+1
-0
docs/source/es/tasks/language_modeling.mdx
docs/source/es/tasks/language_modeling.mdx
+1
-0
No files found.
docs/source/en/tasks/language_modeling.mdx
View file @
bfcd5743
...
...
@@ -141,6 +141,7 @@ Now you need a second preprocessing function to capture text truncated from any
>>> def group_texts(examples):
... concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
... total_length = len(concatenated_examples[list(examples.keys())[0]])
... total_length = (total_length // block_size) * block_size
... result = {
... k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
... for k, t in concatenated_examples.items()
...
...
docs/source/es/tasks/language_modeling.mdx
View file @
bfcd5743
...
...
@@ -141,6 +141,7 @@ Ahora necesitas una segunda función de preprocesamiento para capturar el texto
>>> def group_texts(examples):
... concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
... total_length = len(concatenated_examples[list(examples.keys())[0]])
... total_length = (total_length // block_size) * block_size
... result = {
... k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
... for k, t in concatenated_examples.items()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment