Skip to content

GitLab

Commit 22a3d81a authored by Mostofa Patwary's avatar Mostofa Patwary
Browse files

removed the function compile_helper

parent f7d96a69
Hide whitespace changes
Inline Side-by-side
Showing with 0 additions and 4 deletions
megatron/data/biencoder_dataset_utils.py
View file @ 22a3d81a
...@@ -125,10 +125,6 @@ def get_block_samples_mapping(block_dataset, title_dataset, data_prefix, num_epo ...@@ -125,10 +125,6 @@ def get_block_samples_mapping(block_dataset, title_dataset, data_prefix, num_epo
print_rank_0(' > building samples index mapping for {} ...'.format( print_rank_0(' > building samples index mapping for {} ...'.format(
name)) name))
# compile/bind the C++ helper code
from megatron.data.dataset_utils import compile_helper
compile_helper()
from megatron.data import helpers from megatron.data import helpers
mapping_array = helpers.build_blocks_mapping( mapping_array = helpers.build_blocks_mapping(
block_dataset.doc_idx, block_dataset.doc_idx,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment