1. 23 Oct, 2023 1 commit
    • Yoach Lacombe's avatar
      Add Seamless M4T model (#25693) · cb45f71c
      Yoach Lacombe authored
      
      
      * first raw commit
      
      * still POC
      
      * tentative convert script
      
      * almost working speech encoder conversion scripts
      
      * intermediate code for encoder/decoders
      
      * add modeling code
      
      * first version of speech encoder
      
      * make style
      
      * add new adapter layer architecture
      
      * add adapter block
      
      * add first tentative config
      
      * add working speech encoder conversion
      
      * base model convert works now
      
      * make style
      
      * remove unnecessary classes
      
      * remove unecessary functions
      
      * add modeling code speech encoder
      
      * rework logics
      
      * forward pass of sub components work
      
      * add modeling codes
      
      * some config modifs and modeling code modifs
      
      * save WIP
      
      * new edits
      
      * same output speech encoder
      
      * correct attention mask
      
      * correct attention mask
      
      * fix generation
      
      * new generation logics
      
      * erase comments
      
      * make style
      
      * fix typo
      
      * add some descriptions
      
      * new state
      
      * clean imports
      
      * add tests
      
      * make style
      
      * make beam search and num_return_sequences>1 works
      
      * correct edge case issue
      
      * correct SeamlessM4TConformerSamePadLayer copied from
      
      * replace ACT2FN relu by nn.relu
      
      * remove unecessary return variable
      
      * move back a class
      
      * change name conformer_attention_mask ->conv_attention_mask
      
      * better nit code
      
      * add some Copied from statements
      
      * small nits
      
      * small nit in dict.get
      
      * rename t2u model -> conditionalgeneration
      
      * ongoing refactoring of structure
      
      * update models architecture
      
      * remove SeamlessM4TMultiModal classes
      
      * add tests
      
      * adapt tests
      
      * some non-working code for vocoder
      
      * add seamlessM4T vocoder
      
      * remove buggy line
      
      * fix some hifigan related bugs
      
      * remove hifigan specifc config
      
      * change
      
      * add WIP tokenization
      
      * add seamlessM4T working tokenzier
      
      * update tokenization
      
      * add tentative feature extractor
      
      * Update converting script
      
      * update working FE
      
      * refactor input_values -> input_features
      
      * update FE
      
      * changes in generation, tokenizer and modeling
      
      * make style and add t2u_decoder_input_ids
      
      * add intermediate outputs for ToSpeech models
      
      * add vocoder to speech models
      
      * update valueerror
      
      * update FE with languages
      
      * add vocoder convert
      
      * update config docstrings and names
      
      * update generation code and configuration
      
      * remove todos and update config.pad_token_id to generation_config.pad_token_id
      
      * move block vocoder
      
      * remove unecessary code and uniformize tospeech code
      
      * add feature extractor import
      
      * make style and fix some copies from
      
      * correct consistency + make fix-copies
      
      * add processor code
      
      * remove comments
      
      * add fast tokenizer support
      
      * correct pad_token_id in M4TModel
      
      * correct config
      
      * update tests and codes  + make style
      
      * make some suggested correstion - correct comments and change naming
      
      * rename some attributes
      
      * rename some attributes
      
      * remove unecessary sequential
      
      * remove option to use dur predictor
      
      * nit
      
      * refactor hifigan
      
      * replace normalize_mean and normalize_var with do_normalize + save lang ids to generation config
      
      * add tests
      
      * change tgt_lang logic
      
      * update generation ToSpeech
      
      * add support import SeamlessM4TProcessor
      
      * fix generate
      
      * make tests
      
      * update integration tests, add option to only return text and update tokenizer fast
      
      * fix wrong function call
      
      * update import and convert script
      
      * update integration tests + update repo id
      
      * correct paths and add first test
      
      * update how new attention masks are computed
      
      * update tests
      
      * take first care of batching in vocoder code
      
      * add batching with the vocoder
      
      * add waveform lengths to model outputs
      
      * make style
      
      * add generate kwargs + forward kwargs of M4TModel
      
      * add docstrings forward methods
      
      * reformate docstrings
      
      * add docstrings t2u model
      
      * add another round of modeling docstrings + reformate speaker_id -> spkr_id
      
      * make style
      
      * fix check_repo
      
      * make style
      
      * add seamlessm4t to toctree
      
      * correct check_config_attributes
      
      * write config docstrings + some modifs
      
      * make style
      
      * add docstrings tokenizer
      
      * add docstrings to processor, fe and tokenizers
      
      * make style
      
      * write first version of model docs
      
      * fix FE + correct FE test
      
      * fix tokenizer + add correct integration tests
      
      * fix most tokenization tests
      
      * make style
      
      * correct most processor test
      
      * add generation tests and fix num_return_sequences > 1
      
      * correct integration tests -still one left
      
      * make style
      
      * correct position embedding
      
      * change numbeams to 1
      
      * refactor some modeling code and correct one test
      
      * make style
      
      * correct typo
      
      * refactor intermediate fnn
      
      * refactor feedforward conformer
      
      * make style
      
      * remove comments
      
      * make style
      
      * fix tokenizer tests
      
      * make style
      
      * correct processor tests
      
      * make style
      
      * correct S2TT integration
      
      * Apply suggestions from Sanchit code review
      Co-authored-by: default avatarSanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
      
      * correct typo
      
      * replace torch.nn->nn + make style
      
      * change Output naming (waveforms -> waveform) and ordering
      
      * nit renaming and formating
      
      * remove return None when not necessary
      
      * refactor SeamlessM4TConformerFeedForward
      
      * nit typo
      
      * remove almost copied from comments
      
      * add a copied from comment and remove an unecessary dropout
      
      * remove inputs_embeds from speechencoder
      
      * remove backward compatibiliy function
      
      * reformate class docstrings for a few components
      
      * remove unecessary methods
      
      * split over 2 lines smthg hard to read
      
      * make style
      
      * replace two steps offset by one step as suggested
      
      * nice typo
      
      * move warnings
      
      * remove useless lines from processor
      
      * make generation non-standard test more robusts
      
      * remove torch.inference_mode from tests
      
      * split integration tests
      
      * enrich md
      
      * rename control_symbol_vocoder_offset->vocoder_offset
      
      * clean convert file
      
      * remove tgt_lang and src_lang from FE
      
      * change generate docstring of ToText models
      
      * update generate docstring of tospeech models
      
      * unify how to deal withtext_decoder_input_ids
      
      * add default spkr_id
      
      * unify tgt_lang for t2u_model
      
      * simplify tgt_lang verification
      
      * remove a todo
      
      * change config docstring
      
      * make style
      
      * simplify t2u_tgt_lang_id
      
      * make style
      
      * enrich/correct comments
      
      * enrich .md
      
      * correct typo in docstrings
      
      * add torchaudio dependency
      
      * update tokenizer
      
      * make style and fix copies
      
      * modify SeamlessM4TConverter with new tokenizer behaviour
      
      * make style
      
      * correct small typo docs
      
      * fix import
      
      * update docs and add requirement to tests
      
      * add convert_fairseq2_to_hf in utils/not_doctested.txt
      
      * update FE
      
      * fix imports and make style
      
      * remove torchaudio in FE test
      
      * add seamless_m4t.md to utils/not_doctested.txt
      
      * nits and change the way docstring dataset is loaded
      
      * move checkpoints from ylacombe/ to facebook/ orga
      
      * refactor warning/error to be in the 119 line width limit
      
      * round overly precised floats
      
      * add stereo audio behaviour
      
      * refactor .md and make style
      
      * enrich docs with more precised architecture description
      
      * readd undocumented models
      
      * make fix-copies
      
      * apply some suggestions
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * correct bug from previous commit
      
      * refactor a parameter allowing to clean the code + some small nits
      
      * clean tokenizer
      
      * make style and fix
      
      * make style
      
      * clean tokenizers arguments
      
      * add precisions for some tests
      
      * move docs from not_tested to slow
      
      * modify tokenizer according to last comments
      
      * add copied from statements in tests
      
      * correct convert script
      
      * correct parameter docstring style
      
      * correct tokenization
      
      * correct multi gpus
      
      * make style
      
      * clean modeling code
      
      * make style
      
      * add copied from statements
      
      * add copied statements
      
      * add support with ASR pipeline
      
      * remove file added inadvertently
      
      * fix docstrings seamlessM4TModel
      
      * add seamlessM4TConfig to OBJECTS_TO_IGNORE due of unconventional markdown
      
      * add seamlessm4t to assisted generation ignored models
      
      ---------
      Co-authored-by: default avatarSanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      cb45f71c
  2. 18 Oct, 2023 2 commits
  3. 16 Oct, 2023 1 commit
  4. 12 Oct, 2023 1 commit
  5. 09 Oct, 2023 1 commit
    • tom white's avatar
      fix typos in idefics.md (#26648) · c7f01bee
      tom white authored
      * fix typos in idefics.md
      
      Two typos found in reviewing this documentation.
      
      1) max_new_tokens=4, is not sufficient to generate "Vegetables" as indicated - you will get only "Veget". (incidentally - some mention of how to select this value might be useful as it seems to change in each example)
      
      2) inputs = processor(prompts, return_tensors="pt").to(device) as inputs need to be on the same device (as they are in all other examples on the page)
      
      * Update idefics.md
      
      Change device to cuda explicitly to match other examples
      c7f01bee
  6. 27 Sep, 2023 2 commits
  7. 22 Sep, 2023 1 commit
  8. 18 Sep, 2023 1 commit
  9. 15 Sep, 2023 1 commit
  10. 14 Sep, 2023 1 commit
    • Jinho Park's avatar
      Add BROS (#23190) · 17fdd354
      Jinho Park authored
      
      
      * add Bros boilerplate
      
      * copy and pasted modeling_bros.py from official Bros repo
      
      * update copyright of bros files
      
      * copy tokenization_bros.py from official repo and update import path
      
      * copy tokenization_bros_fast.py from official repo and update import path
      
      * copy configuration_bros.py from official repo and update import path
      
      * remove trailing period in copyright line
      
      * copy and paste bros/__init__.py from official repo
      
      * save formatting
      
      * remove unused unnecessary pe_type argument - using only crel type
      
      * resolve import issue
      
      * remove unused model classes
      
      * remove unnecessary tests
      
      * remove unused classes
      
      * fix original code's bug - layer_module's argument order
      
      * clean up modeling auto
      
      * add bbox to prepare_config_and_inputs
      
      * set temporary value to hidden_size (32 is too low because of the of the
      Bros' positional embedding)
      
      * remove decoder test, update create_and_check* input arguemnts
      
      * add missing variable to model tests
      
      * do make fixup
      
      * update bros.mdx
      
      * add boilerate plate for no_head inference test
      
      * update BROS_PRETRAINED_MODEL_ARCHIVE_LIST (add naver-clova-ocr prefix)
      
      * add prepare_bros_batch_inputs function
      
      * update modeling_common to add bbox inputs in Bros Model Test
      
      * remove unnecessary model inference
      
      * add test case
      
      * add model_doc
      
      * add test case for token_classification
      
      * apply fixup
      
      * update modeling code
      
      * update BrosForTokenClassification loss calculation logic
      
      * revert logits preprocessing logic to make sure logits have original shape
      
      * - update class name
      
      * - add BrosSpadeOutput
      - update BrosConfig arguments
      
      * add boilerate plate for no_head inference test
      
      * add prepare_bros_batch_inputs function
      
      * add test case
      
      * add test case for token_classification
      
      * update modeling code
      
      * update BrosForTokenClassification loss calculation logic
      
      * revert logits preprocessing logic to make sure logits have original shape
      
      * apply masking on the fly
      
      * add BrosSpadeForTokenLinking
      
      * update class name
      put docstring to the beginning of the file
      
      * separate the logits calculation logic and loss calculation logic
      
      * update logic for loss calculation so that logits shape doesn't change
      when return
      
      * update typo
      
      * update prepare_config_and_inputs
      
      * update dummy node initialization
      
      * update last_hidden_states getting logic to consider when return_dict is False
      
      * update box first token mask param
      
      * bugfix: remove random attention mask generation
      
      * update keys to ignore on load missing
      
      * run make style and quality
      
      * apply make style and quality of other codes
      
      * update box_first_token_mask to bool type
      
      * update index.md
      
      * apply make style and quality
      
      * apply make fix-copies
      
      * pass check_repo
      
      * update bros model doc
      
      * docstring bugfix fix
      
      * add checkpoint for doc, tokenizer for doc
      
      * Update README.md
      
      * Update docs/source/en/model_doc/bros.md
      Co-authored-by: default avataramyeroberts <22614925+amyeroberts@users.noreply.github.com>
      
      * Update bros.md
      
      * Update src/transformers/__init__.py
      Co-authored-by: default avataramyeroberts <22614925+amyeroberts@users.noreply.github.com>
      
      * Update docs/source/en/model_doc/bros.md
      Co-authored-by: default avataramyeroberts <22614925+amyeroberts@users.noreply.github.com>
      
      * Apply suggestions from code review
      Co-authored-by: default avataramyeroberts <22614925+amyeroberts@users.noreply.github.com>
      
      * apply suggestions from code review
      
      * apply suggestions from code review
      
      * revert test_processor_markuplm.py
      
      * Update test_processor_markuplm.py
      
      * apply suggestions from code review
      
      * apply suggestions from code review
      
      * apply suggestions from code review
      
      * update BrosSpadeELForTokenClassification head name to entity linker
      
      * add doc string for config params
      
      * update class, var names to more explicit and apply suggestions from code review
      
      * remove unnecessary keys to ignore
      
      * update relation extractor to be initialized with config
      
      * add bros processor
      
      * apply make style and quality
      
      * update bros.md
      
      * remove bros tokenizer, add bros processor that wraps bert tokenizer
      
      * revert change
      
      * apply make fix-copies
      
      * update processor code, update itc -> initial token, stc -> subsequent token
      
      * add type hint
      
      * remove unnecessary condition branches in embedding forward
      
      * fix auto tokenizer fail
      
      * update docstring for each classes
      
      * update bbox input dimension as standard 2 points and convert them to 4
      points in forward pass
      
      * update bros docs
      
      * apply suggestions from code review : update Bros -> BROS in bros.md
      
      * 1. box prefix var -> bbox
      2. update variable names to be more explicit
      
      * replace einsum with torch matmul
      
      * apply style and quality
      
      * remove unused argument
      
      * remove unused arguments
      
      * update docstrings
      
      * apply suggestions from code review: add BrosBboxEmbeddings, replace
      einsum with classical matrix operations
      
      * revert einsum update
      
      * update bros processor
      
      * apply suggestions from code review
      
      * add conversion script for bros
      
      * Apply suggestions from code review
      
      * fix readme
      
      * apply fix-copies
      
      ---------
      Co-authored-by: default avataramyeroberts <22614925+amyeroberts@users.noreply.github.com>
      17fdd354
  11. 12 Sep, 2023 2 commits
  12. 25 Aug, 2023 1 commit
    • Arthur's avatar
      [`CodeLlama`] Add support for `CodeLlama` (#25740) · 015f8e11
      Arthur authored
      
      
      * add all
      
      * Revert "Delete .github directory"
      
      This reverts commit 9b0ff7b052e2b20b629a26fb13606b78a42944d1.
      
      * make conversion script backward compatible
      
      * fixup
      
      * more styling
      
      * copy to llama changes
      
      * fix repo consistency
      
      * nits
      
      * document correct classes
      
      * updates
      
      * more fixes
      
      * nits
      
      * update auto mappings
      
      * add readmes
      
      * smallupdates
      
      * llama-code replace with llama_code
      
      * make fixup
      
      * updates to the testsing suite
      
      * fix fast nits
      
      * more small fixes
      
      * fix decode
      
      * fix template processing
      
      * properly reset the normalizer
      
      * nits processor
      
      * tokenization tests pass
      
      * styling
      
      * last tests
      
      * additional nits
      
      * one test is left
      
      * nits
      
      Co-authored-by faabian <faabian@users.noreply.github.com>
      
      * update failing test
      
      * fixup
      
      * remove decode infilling users should handle it on their onw after generation, padding can be a problem
      
      * update
      
      * make test slow and more meaningfull
      
      * fixup
      
      * doc update
      
      * fixup
      
      * Apply suggestions from code review
      
      * add kwargs doc
      
      * tokenizer requires `requires_backend`
      
      * type requires_backends
      
      * CodeLlama instead of LlamaCode
      
      * more name cahnges
      
      * nits
      
      * make doctests happy
      
      * small pipeline nits
      
      * last nit
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * update
      
      * add codellama to toctree
      
      ---------
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      015f8e11
  13. 18 Aug, 2023 1 commit
    • Stas Bekman's avatar
      new model: IDEFICS via HuggingFaceM4 (#24796) · 6c811a32
      Stas Bekman authored
      
      
      * rename
      
      * restore
      
      * mappings
      
      * unedited tests+docs
      
      * docs
      
      * fixes
      
      * fix auto-sync breakage
      
      * cleanup
      
      * wip
      
      * wip
      
      * add fetch_images
      
      * remove einops dependency
      
      * update
      
      * fix
      
      * fix
      
      * fix
      
      * fix
      
      * fix
      
      * re-add
      
      * add batching
      
      * rework
      
      * fix
      
      * improve
      
      * add Leo as I am extending his work
      
      * cleanup
      
      * fix
      
      * cleanup
      
      * slow-test
      
      * fix
      
      * fix
      
      * fixes
      
      * deal with warning
      
      * rename modified llama classes
      
      * rework fetch_images
      
      * alternative implementation
      
      * cleanup
      
      * strict version
      
      * cleanup
      
      * [`IDEFICS`]聽Fix idefics ci (#25056)
      
      * Fix IDEFICS CI
      
      * fix test file
      
      * fixup
      
      * some changes to make tests pass
      
      * fix
      
      * fixup
      
      * Update src/transformers/models/idefics/configuration_idefics.py
      Co-authored-by: default avatarStas Bekman <stas00@users.noreply.github.com>
      
      ---------
      Co-authored-by: default avatarStas Bekman <stas00@users.noreply.github.com>
      
      * remove compat checks
      
      * style
      
      * explain that Idefics is not for training from scratch
      
      * require pt>=2.0
      
      * fix idefics vision config (#25092)
      
      * fix idefics vision config
      
      * fixup
      
      * clean
      
      * Update src/transformers/models/idefics/configuration_idefics.py
      
      ---------
      Co-authored-by: default avatarStas Bekman <stas00@users.noreply.github.com>
      
      * cleanup
      
      * style
      
      * cleanup
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * upcase
      
      * sequence of images
      
      * handle the case with no images
      
      * Update src/transformers/image_processing_utils.py
      Co-authored-by: default avatarVictor SANH <victorsanh@gmail.com>
      
      * support pure lm take 2
      
      * support tokenizer options
      
      * parameterize num_channels
      
      * fix upcase
      
      * s|IdeficsForCausalLM|IdeficsForVisionText2Text|g
      
      * manual to one line
      
      * addressing review
      
      * unbreak
      
      * remove clip dependency
      
      * fix test
      
      * consistency
      
      * PIL import
      
      * Idefics prefix
      
      * Idefics prefix
      
      * hack to make tests work
      
      * style
      
      * fix
      
      * fix
      
      * revert
      
      * try/finally
      
      * cleanup
      
      * clean up
      
      * move
      
      * [`IDEFICS`] Fix idefics config refactor (#25149)
      
      * refactor config
      
      * nuke init weights
      
      * more refactor
      
      * oops
      
      * remove visual question answering pipeline support
      
      * Update src/transformers/models/idefics/clip.py
      Co-authored-by: default avatarStas Bekman <stas00@users.noreply.github.com>
      
      * Update src/transformers/models/idefics/modeling_idefics.py
      
      * cleanup
      
      * mv clip.py vision.py
      
      * tidyup
      
      ---------
      Co-authored-by: default avatarStas Bekman <stas00@users.noreply.github.com>
      Co-authored-by: default avatarStas Bekman <stas@stason.org>
      
      * fix
      
      * license
      
      * condition on pt
      
      * fix
      
      * style
      
      * fix
      
      * rm torchvision dependency, allow custom transforms
      
      * address review
      
      * rework device arg
      
      * add_eos_token
      
      * s/transforms/transform/
      
      * fix top level imports
      
      * fix return value
      
      * cleanup
      
      * cleanup
      
      * fix
      
      * style
      
      * license
      
      * license
      
      * Update src/transformers/models/idefics/image_processing_idefics.py
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * add a wrapper to freeze vision layears
      
      * tidyup
      
      * use the correct std/mean settings
      
      * parameterize values from config
      
      * add tests/models/idefics/test_image_processing_idefics.py
      
      * add test_processor_idefics.py
      
      * cleanup
      
      * cleanups
      
      * fix
      
      * fix
      
      * move to the right group
      
      * style
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * add perceiver config
      
      * reset
      
      * missing arg docs
      
      * Apply suggestions from code review
      Co-authored-by: default avatarLeo Tronchon <leo.tronchon@gmail.com>
      
      * address review comments
      
      * inject automatic end of utterance tokens (#25218)
      
      * inject automatic end of utterance tokens
      
      * fix
      
      * fix
      
      * fix
      
      * rework to not use the config
      
      * not end_of_utterance_token at the end
      
      * Update src/transformers/models/idefics/processing_idefics.py
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * address review
      
      * Apply suggestions from code review
      Co-authored-by: default avatarJoao Gante <joaofranciscocardosogante@gmail.com>
      
      * Update src/transformers/image_processing_utils.py
      Co-authored-by: default avatarNicolas Patry <patry.nicolas@protonmail.com>
      
      * [`Idefics`] add image_embeddings option in generate-related methods (#25442)
      
      * add image_embeddings option in generate-related methods
      
      * style
      
      * rename image_embeddings and allow perceiver embeddings precomputation
      
      * compute embeddings within generate
      
      * make is_encoder_decoder= True the default in config
      
      * nested if else fix
      
      * better triple check
      
      * switch if elif order for pixel values / img embeds
      
      * update model_kwargs perceiver only at the end
      
      * use _prepare_model_inputs instead of encoder_decoder logic
      
      * fix comment typo
      
      * fix config default for is_encoder_decoder
      
      * style
      
      * add typehints
      
      * precompute in forward
      
      * doc builder
      
      * style
      
      * pop instead of get image hidden states
      
      * Trigger CI
      
      * Update src/transformers/models/idefics/modeling_idefics.py
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * Update src/transformers/models/idefics/modeling_idefics.py
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * fix * + indentation + style
      
      * simplify a bit the use_resampler logic using comments
      
      * update diocstrings
      
      * Trigger CI
      
      ---------
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * fix rebase changes
      
      * unbreak #25237 - to be fixed in follow up PRs
      
      * is_composition = False
      
      * no longer needed
      
      ---------
      Co-authored-by: default avatarleot13 <leo.tronchon@gmail.com>
      Co-authored-by: default avatarYounes Belkada <49240599+younesbelkada@users.noreply.github.com>
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      Co-authored-by: default avatarVictor SANH <victorsanh@gmail.com>
      Co-authored-by: default avatarJoao Gante <joaofranciscocardosogante@gmail.com>
      Co-authored-by: default avatarNicolas Patry <patry.nicolas@protonmail.com>
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      6c811a32
  14. 13 Aug, 2023 1 commit
  15. 09 Aug, 2023 1 commit
  16. 25 Jul, 2023 2 commits
    • Sebastian Husch Lee's avatar
      [`T5`, `MT5`, `UMT5`] Add [T5, MT5, UMT5]ForSequenceClassification (#24726) · 8f36ab3e
      Sebastian Husch Lee authored
      * Initial addition of t5forsequenceclassification
      
      * Adding imports and adding tests
      
      * Formatting
      
      * Running make fix-copies
      
      * Adding mt5forseq
      
      * Formatting
      
      * run make fix-copies
      
      * Adding to docs
      
      * Add model_parallel
      
      * Fix bug
      
      * Fix
      
      * Remove TODO
      
      * Fixing tests for T5ForSequenceClassification
      
      * Undo changes to dependency_versions_table.py
      
      * Change classification head to work with T5Config directly
      
      * Change seq length to let tests pass
      
      * PR comments for formatting
      
      * Formatting
      
      * Initial addition of UMT5ForSequenceClassification
      
      * Adding to inits and formatting
      
      * run make fix-copies
      
      * Add doc for UMT5ForSeqClass
      
      * Update UMT5 config
      
      * Fix docs
      
      * Skip torch fx test for SequenceClassification
      
      * Formatting
      
      * Add skip to UMT5 tests as well
      
      * Fix umt5 tests
      
      * Running make fix-copies
      
      * PR comments
      
      * Fix for change to sentence_representation
      
      * Rename seq_len to hidden_size since that's what it is
      
      * Use base_model to follow format of the rest of the library
      
      * Update docs
      
      * Extract the decoder_input_ids changes and make one liner
      
      * Make one-liner
      8f36ab3e
    • Arthur's avatar
      [`MPT`] Add MosaicML's `MPT` model to transformers (#24629) · dcb183f4
      Arthur authored
      
      
      * draft add new model like
      
      * some cleaning of the config
      
      * nits
      
      * add nested configs
      
      * nits
      
      * update
      
      * update
      
      * added layer norms + triton kernels
      
      * consider only LPLayerNorm for now.
      
      * update
      
      * all keys match.
      
      * Update
      
      * fixing nits here and there
      
      * working forward pass.
      
      * removed einops dependency
      
      * nits
      
      * format
      
      * add alibi
      
      * byebye head mask
      
      * refactor attention
      
      * nits.
      
      * format
      
      * fix nits.
      
      * nuke ande updates
      
      * nuke tokenizer test
      
      * don't reshape query with kv heads
      
      * added a bit of documentation.
      
      * remove unneeded things
      
      * nuke more stuff
      
      * nit
      
      * logits match - same generations
      
      * rm unneeded methods
      
      * 1 remaining failing CI test
      
      * nit
      
      * fix nits
      
      * fix docs
      
      * fix docs
      
      * rm tokenizer
      
      * fixup
      
      * fixup
      
      * fixup and fix tests
      
      * fixed configuration object.
      
      * use correct activation
      
      * few minor fixes
      
      * clarify docs a bit
      
      * logits match 脿 1e-12
      
      * skip and unskip a test
      
      * added some slow tests.
      
      * fix readme
      
      * add more details
      
      * Update docs/source/en/model_doc/mpt.md
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * Apply suggestions from code review
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * fix configuration issues
      
      * more fixes in config
      
      * added more models
      
      * Apply suggestions from code review
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * remove unneeded position ids
      
      * fix some  comments
      
      * Apply suggestions from code review
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * revert suggestion
      
      * mpt alibi + added batched generation
      
      * Update src/transformers/models/mpt/__init__.py
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * remove init config
      
      * Update src/transformers/models/mpt/configuration_mpt.py
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * fix nit
      
      * add another slow test
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * fits in one line
      
      * some refactor because make fixup doesn't pass
      
      * add ft notebook
      
      * update md
      
      * correct doc path
      
      ---------
      Co-authored-by: default avataryounesbelkada <younesbelkada@gmail.com>
      Co-authored-by: default avatarYounes Belkada <49240599+younesbelkada@users.noreply.github.com>
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      dcb183f4
  17. 24 Jul, 2023 1 commit
    • Rinat's avatar
      Pvt model (#24720) · a03d13c8
      Rinat authored
      * pull and push updates
      
      * add docs
      
      * fix modeling
      
      * Add and run test
      
      * make copies
      
      * add task
      
      * fix tests and fix small issues
      
      * Checks on a Pull Request
      
      * fix docs
      
      * add desc pvt.md
      a03d13c8
  18. 18 Jul, 2023 1 commit
    • NielsRogge's avatar
      Add DINOv2 (#24016) · 3ec10e6c
      NielsRogge authored
      * First draft
      
      * More improvements
      
      * Convert patch embedding layer
      
      * Convert all weights
      
      * Make conversion work
      
      * Improve conversion script
      
      * Fix style
      
      * Make all tests pass
      
      * Add image processor to auto mapping
      
      * Add swiglu ffn
      
      * Add image processor to conversion script
      
      * Fix conversion of giant model
      
      * Fix documentation
      
      * Fix style
      
      * Fix tests
      
      * Address comments
      
      * Address more comments
      
      * Remove unused arguments
      
      * Remove more arguments
      
      * Rename parameters
      
      * Include mask token
      
      * Address comments
      
      * Add docstring
      
      * Transfer checkpoints
      
      * Empty commit
      3ec10e6c
  19. 11 Jul, 2023 2 commits
  20. 10 Jul, 2023 1 commit
  21. 05 Jul, 2023 1 commit
  22. 03 Jul, 2023 1 commit
    • Arthur's avatar
      [`Umt5`] Add google's umt5 to `transformers` (#24477) · 799df10a
      Arthur authored
      
      
      * add tokenization template
      
      * update conversion script
      
      * update modeling code
      
      * update
      
      * update convert checkpoint
      
      * update modeling
      
      * revert changes on convert script
      
      * new conversion script for new format
      
      * correct position bias
      
      * cleaning a bit
      
      * Credit co authors
      Co-authored-by: default avataragemagician <ahmed.elnaggar@tum.de>
      
      Co-authored-by: stefan-it
      <>
      
      * styling
      
      * Add docq
      
      * fix copies
      
      * add co author
      
      * Other Author
      
      * Merge branch 'main' of https://github.com/huggingface/transformers
      
       into add-umt5
      
      * add testing
      
      * nit
      
      * Update docs/source/en/model_doc/umt5.mdx
      Co-authored-by: default avatarStefan Schweter <stefan@schweter.it>
      
      * fix t5
      
      * actual fix?
      
      * revert wrong changes
      
      * remove
      
      * update test
      
      * more fixes
      
      * revert some changes
      
      * add SPIECE_UNDERLINE
      
      * add a commone xample
      
      * upfate
      
      * fix copies
      
      * revert changes on t5 conversion script
      
      * revert bytefallback changes since there was no addition yet
      
      * fixup
      
      * fixup
      
      * ingore umt5 cutom testing folder
      
      * fix readmes
      
      * revertT5 changes
      
      * same outputs
      
      * fixup
      
      * update example
      
      * Apply suggestions from code review
      
      * style
      
      * draft addition of all new files
      
      * current update
      
      * fix attention and stuff
      
      * finish refactoring
      
      * auto config
      
      * fixup
      
      * more nits
      
      * add umt5 to init
      
      * use md format
      
      * Update README.md
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * revert changes on mt5
      
      * revert mt4 changes
      
      * update test
      
      * more fixes
      
      * add to mapping
      
      * fix-copies
      
      * fix copies
      
      * foix retain grad
      
      * fix some tests
      
      * nits
      
      * done
      
      * Update src/transformers/models/umt5/modeling_umt5.py
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * Update docs/source/en/model_doc/umt5.md
      
      * Update src/transformers/models/umt5/__init__.py
      
      * Update docs/source/en/model_doc/umt5.md
      Co-authored-by: default avatarStefan Schweter <stefan@schweter.it>
      
      * Update src/transformers/models/umt5/modeling_umt5.py
      
      * update conversion script + use google checkpoints
      
      * nits
      
      * update test and modelling
      
      * stash slow convert
      
      * update fixupd
      
      * don't change slow
      
      ---------
      
      Co-authored-by: stefan-it <>
      Co-authored-by: default avatarStefan Schweter <stefan@schweter.it>
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      799df10a
  23. 29 Jun, 2023 3 commits
    • amyeroberts's avatar
      Removal of deprecated vision methods and specify deprecation versions (#24570) · b324557a
      amyeroberts authored
      * Removal of deprecated methods and specify versions
      
      * Fix tests
      b324557a
    • Sanchit Gandhi's avatar
      Add Musicgen (#24109) · 1c1c9075
      Sanchit Gandhi authored
      
      
      * Add Audiocraft
      
      * add cross attention
      
      * style
      
      * add for lm
      
      * convert and verify
      
      * introduce t5
      
      * split configs
      
      * load t5 + lm
      
      * clean conversion
      
      * copy from t5
      
      * style
      
      * start pattern provider
      
      * make generation work
      
      * style
      
      * fix pos embs
      
      * propagate shape changes
      
      * propagate shape changes
      
      * style
      
      * delay pattern: pad tokens at end
      
      * audiocraft -> musicgen
      
      * fix inits
      
      * add mdx
      
      * style
      
      * fix pad token in processor
      
      * override generate and add todos
      
      * add init to test
      
      * undo pattern delay mask after gen
      
      * remove cfg logits processor
      
      * remove cfg logits processor
      
      * remove logits processor in favour of mask
      
      * clean pos embs
      
      * make fix copies
      
      * update readmes
      
      * clean pos emb
      
      * refactor encoder/decoder
      
      * make fix copies
      
      * update conversion
      
      * fix config imports
      
      * update config docs
      
      * make style
      
      * send pattern mask to device
      
      * pattern mask with delay
      
      * recover prompted audio tokens
      
      * fix docstrings
      
      * laydown test file
      
      * pattern edge case
      
      * remove t5 ref
      
      * add processing class
      
      * config refactor
      
      * better pattern comment
      
      * check if mask is not present
      
      * check if mask is not present
      
      * refactor to auto class
      
      * remove encoder configs
      
      * fix processor
      
      * processor import
      
      * start updating conversion
      
      * start updating tests
      
      * make style
      
      * convert t5, encodec, lm
      
      * convert as composite
      
      * also convert processor
      
      * run generate
      
      * classifier free gen
      
      * comments and clean up
      
      * make style
      
      * docs for logit proc
      
      * docstring for uncond gen
      
      * start lm tests
      
      * work tests
      
      * let the lm generate
      
      * refactor: reshape inside forward
      
      * undo greedy loop changes
      
      * from_enc_dec -> from_sub_model
      
      * fix input id shapes in docstrings
      
      * Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * undo generate changes
      
      * from sub model config
      
      * Update src/transformers/models/musicgen/modeling_musicgen.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * make generate work again
      
      * generate uncond -> get uncond inputs
      
      * remove prefix allowed tokens fn
      
      * better error message
      
      * logit proc checks
      
      * Apply suggestions from code review
      Co-authored-by: default avatarJoao Gante <joaofranciscocardosogante@gmail.com>
      
      * make decoder only tests work
      
      * composite fast tests
      
      * make style
      
      * uncond generation
      
      * feat extr padding
      
      * make audio prompt work
      
      * fix inputs docstrings
      
      * unconditional inputs: dict -> model output
      
      * clean up tests
      
      * more clean up tests
      
      * make style
      
      * t5 encoder -> auto text encoder
      
      * remove comments
      
      * deal with frames
      
      * fix auto text
      
      * slow tests
      
      * nice mdx
      
      * remove can generate
      
      * todo - hub id
      
      * convert m/l
      
      * make fix copies
      
      * only import generation with torch
      
      * ignore decoder from tests
      
      * don't wrap uncond inputs
      
      * make style
      
      * cleaner uncond inputs
      
      * add example to musicgen forward
      
      * fix docs
      
      * ignore MusicGen Model/ForConditionalGeneration in auto mapping
      
      * add doc section to toctree
      
      * add to doc tests
      
      * add processor tests
      
      * fix push to hub in conversion
      
      * tips for decoder only loading
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      
      * fix conversion for s / m / l checkpoints
      
      * import stopping criteria from module
      
      * remove from pipeline tests
      
      * fix uncond docstring
      
      * decode audio method
      
      * fix docs
      
      * org: sanchit-gandhi -> facebook
      
      * fix max pos embeddings
      
      * remove auto doc (not compatible with shapes)
      
      * bump max pos emb
      
      * make style
      
      * fix doc
      
      * fix config doc
      
      * fix config doc
      
      * ignore musicgen config from docstring
      
      * make style
      
      * fix config
      
      * fix config for doctest
      
      * consistent from_sub_models
      
      * don't automap decoder
      
      * fix mdx save audio file
      
      * fix mdx save audio file
      
      * processor batch decode for audio
      
      * remove keys to ignore
      
      * update doc md
      
      * update generation config
      
      * allow changes for default generation config
      
      * update tests
      
      * make style
      
      * fix docstring for uncond
      
      * fix processor test
      
      * fix processor test
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarJoao Gante <joaofranciscocardosogante@gmail.com>
      Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
      1c1c9075
    • amyeroberts's avatar
      Update old existing feature extractor references (#24552) · ae454f41
      amyeroberts authored
      * Update old existing feature extractor references
      
      * Typo
      
      * Apply suggestions from code review
      
      * Apply suggestions from code review
      
      * Apply suggestions from code review
      
      * Address comments from review - update 'feature extractor'
      Co-authored by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
      ae454f41
  24. 28 Jun, 2023 1 commit
  25. 27 Jun, 2023 1 commit
  26. 26 Jun, 2023 1 commit
  27. 20 Jun, 2023 1 commit
  28. 19 Jun, 2023 1 commit
  29. 14 Jun, 2023 1 commit
  30. 09 Jun, 2023 1 commit
  31. 02 Jun, 2023 2 commits
  32. 24 May, 2023 1 commit