"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "fc7ce2ebc52eccd8158a7feeeee11eb44f964937"
Use text_column_name variable instead of "text" (#12132)
* Use text_column_name variable instead of "text"
`text_column_name` was already defined above where I made the changes and it was also used below where I made changes.
This is a very minor change. If a dataset does not use "text" as the column name, then the `tokenize_function` will now use whatever column is assigned to `text_column_name`. `text_column_name` is just the first column name if "text" is not a column name. It makes the function a little more robust, though I would assume that 90% + of datasets use "text" anyway.
* black formatting
* make style
Co-authored-by:
Nicholas Broad <nicholas@nmbroad.com>
Showing
Please register or sign in to comment