GitLab

Modify 'unambigiously' to 'unambiguously'

If the value of rand_end is returned from the randint function, the value of sampled_doc_index that matches current_idx is returned from searchsorted.

example:
cumsum_max = {int64} 30
doc_cumsum = {ndarray} [ 5  7 11 19 30]
doc_lengths = {list} <class 'list'>: [5, 2, 4, 8, 11]
if current_idx  = 1,
rand_start = 7
rand_end = 35
sentence_index = randint(7, 35) % cumsum_max
if randint return 35, sentence_index becomes 5.
if sentence_index is 5, np.searchsorted returns 1 equal to current_index.

The unconditional generation works now but if the seed is fixed, the sample is the same every time.
n_samples > 1 will give different samples though.
I am giving the start token as '<|endoftext|>' for the unconditional generation.

order.

data on disc as a memmap rather than in memory

data on disc as a memmap rather than in memory

out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.

out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.

Maybe better.

modify `heruistic` to `heuristic` in line 660, `charcter` to `character` in line 661.

modify `mull` to `null` in line 474 annotation.

delete redundancy line `if args.train`, simplify code.