Skip to content

GitLab

Commit 01bcbb1e authored by lim's avatar lim
Browse files

Initial commit

parent 187361d1
Pipeline #3395 canceled with stages
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 118 additions and 0 deletions
data/oscar-1GB_head-qwen_text_document/cache/GPTDataset_indices/961e8fd704df44e55eba1243e590da39-GPTDataset-test-description.txt 0 → 100644
View file @ 01bcbb1e
{
"class": "GPTDataset",
"dataset_path": "/workspace/data/oscar/oscar-1GB_head-qwen_text_document",
"num_samples": 160,
"index_split": "test",
"random_seed": 1234,
"sequence_length": 4096,
"split": "949,50,1",
"split_matrix": [
[
0,
0.949
],
[
0.949,
0.999
],
[
0.999,
1.0
]
],
"tokenizer": {
"class": "megatron.core.tokenizers.text.models.default_tokenizer.DefaultTokenizerText",
"tokenizer_path": "/home/models/qwen3/Qwen3-8B",
"vocab_file": "None",
"merges_file": "None"
}
}
\ No newline at end of file
data/oscar-1GB_head-qwen_text_document/cache/GPTDataset_indices/a253c4536f250d0e6a93787757faacb4-GPTDataset-test-description.txt 0 → 100644
View file @ 01bcbb1e
{
"class": "GPTDataset",
"dataset_path": "/workspace/data/oscar/oscar-1GB_head-qwen_text_document",
"num_samples": 160,
"index_split": "test",
"random_seed": 1234,
"sequence_length": 32768,
"split": "949,50,1",
"split_matrix": [
[
0,
0.949
],
[
0.949,
0.999
],
[
0.999,
1.0
]
],
"tokenizer": {
"class": "megatron.core.tokenizers.text.models.default_tokenizer.DefaultTokenizerText",
"tokenizer_path": "/home/models/qwen3/Qwen3-8B",
"vocab_file": "None",
"merges_file": "None"
}
}
\ No newline at end of file
data/oscar-1GB_head-qwen_text_document/cache/GPTDataset_indices/aa0a9cea3af4e48ffec2635ce0475149-GPTDataset-valid-description.txt 0 → 100644
View file @ 01bcbb1e
{
"class": "GPTDataset",
"dataset_path": "/workspace/data/oscar/oscar-1GB_head-qwen_text_document",
"num_samples": 160,
"index_split": "valid",
"random_seed": 1234,
"sequence_length": 8192,
"split": "949,50,1",
"split_matrix": [
[
0,
0.949
],
[
0.949,
0.999
],
[
0.999,
1.0
]
],
"tokenizer": {
"class": "megatron.core.tokenizers.text.models.default_tokenizer.DefaultTokenizerText",
"tokenizer_path": "/home/models/qwen3/Qwen3-8B",
"vocab_file": "None",
"merges_file": "None"
}
}
\ No newline at end of file
data/oscar-1GB_head-qwen_text_document/cache/GPTDataset_indices/e37cd1379baa5fb23bb48f5ddc502b25-GPTDataset-train-description.txt 0 → 100644
View file @ 01bcbb1e
{
"class": "GPTDataset",
"dataset_path": "/workspace/data/oscar/oscar-1GB_head-qwen_text_document",
"num_samples": 12800,
"index_split": "train",
"random_seed": 1234,
"sequence_length": 4096,
"split": "949,50,1",
"split_matrix": [
[
0,
0.949
],
[
0.949,
0.999
],
[
0.999,
1.0
]
],
"tokenizer": {
"class": "megatron.core.tokenizers.text.models.default_tokenizer.DefaultTokenizerText",
"tokenizer_path": "/home/models/qwen3/Qwen3-0.6B",
"vocab_file": "None",
"merges_file": "None"
}
}
\ No newline at end of file
data/vocab.json 0 → 100644
View file @ 01bcbb1e
This diff is collapsed.
dcu_megatron/__init__.py 0 → 100644
View file @ 01bcbb1e
from .adaptor import megatron_adaptor
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment