Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
Megatron-LM
Commits
64aef3a5
Commit
64aef3a5
authored
Apr 13, 2020
by
Neel Kant
Browse files
Debug helpers compilation
parent
27ecc17a
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
7 deletions
+2
-7
megatron/data/helpers.cpp
megatron/data/helpers.cpp
+2
-7
No files found.
megatron/data/helpers.cpp
View file @
64aef3a5
...
@@ -361,11 +361,6 @@ py::array build_blocks_mapping_impl(const py::array_t<int64_t>& docs_,
...
@@ -361,11 +361,6 @@ py::array build_blocks_mapping_impl(const py::array_t<int64_t>& docs_,
// Set the flag on second iteration.
// Set the flag on second iteration.
second
=
(
iteration
==
1
);
second
=
(
iteration
==
1
);
// Counters:
uint64_t
empty_docs
=
0
;
uint64_t
one_sent_docs
=
0
;
uint64_t
long_sent_docs
=
0
;
// Current map index.
// Current map index.
uint64_t
map_index
=
0
;
uint64_t
map_index
=
0
;
...
@@ -384,7 +379,7 @@ py::array build_blocks_mapping_impl(const py::array_t<int64_t>& docs_,
...
@@ -384,7 +379,7 @@ py::array build_blocks_mapping_impl(const py::array_t<int64_t>& docs_,
// Document sentences are in [sent_index_first, sent_index_last)
// Document sentences are in [sent_index_first, sent_index_last)
const
auto
sent_index_first
=
docs
[
doc
];
const
auto
sent_index_first
=
docs
[
doc
];
const
auto
sent_index_last
=
docs
[
doc
+
1
];
const
auto
sent_index_last
=
docs
[
doc
+
1
];
const
auto
target_seq_len
=
max_seq_length
-
titles_sizes
[
doc
]
const
auto
target_seq_len
=
max_seq_length
-
titles_sizes
[
doc
]
;
// At the begining of the document previous index is the
// At the begining of the document previous index is the
// start index.
// start index.
...
@@ -515,5 +510,5 @@ py::array build_blocks_mapping(const py::array_t<int64_t>& docs_,
...
@@ -515,5 +510,5 @@ py::array build_blocks_mapping(const py::array_t<int64_t>& docs_,
PYBIND11_MODULE
(
helpers
,
m
)
{
PYBIND11_MODULE
(
helpers
,
m
)
{
m
.
def
(
"build_mapping"
,
&
build_mapping
);
m
.
def
(
"build_mapping"
,
&
build_mapping
);
m
.
def
(
"build_blocks_mapping"
,
&
build_blocks_mapping
)
m
.
def
(
"build_blocks_mapping"
,
&
build_blocks_mapping
)
;
}
}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment