Skip to content

GitLab

Commit 17b04444 authored by cardy20's avatar cardy20
Browse files

conflict changed

parent c69f6c38
Show whitespace changes
Inline Side-by-side
Showing with 0 additions and 36 deletions
get_ngram.sh deleted 100644 → 0
View file @ c69f6c38
export PYTHONPATH=$PWD
python3 scripts/clean_training_data/generate_13_grams.py \
-dir /fsx/polyglot/massivetext_large_data/ \
-sdir /fsx/lime12/ngram_train2/ -n 13 -buckets 500
get_ngram2.sh deleted 100644 → 0
View file @ c69f6c38
export PYTHONPATH=$PWD
python3 scripts/clean_training_data/generate_13_grams.py \
-dir /fsx/kevinai/data/ko/merged_raw/ \
-sdir /fsx/lime12/ngram_merged_raw -n 13 -buckets 500
\ No newline at end of file
ngrams.log deleted 100644 → 0
View file @ c69f6c38
INFO - 05/29/23 02:24:05 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:24:05 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 02:26:29 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:26:29 - 0:00:00 - Starting at pile document index 106000
INFO - 05/29/23 02:29:19 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:29:19 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 02:31:50 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:31:50 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 02:32:22 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:32:22 - 0:00:00 - ngrams already generated and bucketed, skipping
INFO - 05/29/23 02:34:01 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:34:01 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 02:34:58 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 02:34:58 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 07:12:33 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 07:12:33 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 07:26:46 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 07:26:46 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 07:30:21 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 07:30:21 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 07:31:54 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 07:31:54 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 13:27:39 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 13:27:39 - 0:00:00 - Starting at pile document index 8432000
INFO - 05/29/23 13:30:28 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 13:30:28 - 0:00:00 - Starting at pile document index 0
INFO - 05/29/23 14:27:00 - 0:00:00 - Generating 13-grams and bucketing.
INFO - 05/29/23 14:27:00 - 0:00:00 - Starting at pile document index 0
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment