• Taras Tsugrii's avatar
    Replace appends with list comprehension. (#23359) · 4e244b88
    Taras Tsugrii authored
    It's more idiomatic and significantly more efficient because
    1) it avoids repeated `append` call that Python has to resolve on each iteration
    2) can preallocate the size of the final list avoiding resizing
    4e244b88
tokenization_utils_fast.py 32.6 KB