Make sure we also change the config when setting `encoder_hid_dim_type=="text_proj"` and allow xformers (#3615)

* fix if

* make style

* make style

* add tests for xformers

* make style

* update