• cody-moveworks's avatar
    Make OpenAIGPTTokenizer work with SpaCy 2.x and 3.x (#15019) · a54961c5
    cody-moveworks authored
    * Make OpenAIGPTTokenizer work with SpaCy 3.x
    
    SpaCy 3.x introduced an API change to creating the tokenizer that
    breaks OpenAIGPTTokenizer. The old API for creating the tokenizer in
    SpaCy 2.x no longer works under SpaCy 3.x, but the new API for creating
    the tokenizer in SpaCy 3.x DOES work under SpaCy 2.x. Switching to the
    new API should allow OpenAIGPTTokenizer to work under both SpaCy 2.x and
    SpaCy 3.x versions.
    
    * Add is_spacy_available and is_ftfy_available methods to file utils
    
    * Add spacy and ftfy unittest decorator to testing utils
    
    * Add tests for OpenAIGPTTokenizer that require spacy and ftfy
    
    * Modify CircleCI config to run tests that require spacy and ftfy
    
    * Remove unneeded unittest decorators are reuse test code
    
    * Run make fixup
    a54961c5
test_tokenization_openai.py 4.9 KB