GitLab

The previous data_dir_to_fasta.py script is very slow and requires fully reparsing mmCIF files. This new script is much faster and uses the sequence information from the alignment data instead. Note that this will not include chains for which alignments could not be generated, but we can't use those during training anyways.

This adds support for duplicate chain expansion for the alignment dir format. This script can be run on the flattened non-redundant RODA alignments to add explicit directories for all of the duplicate chains in the duplicate_chains file, symlinked to their representative chain alignment directory.

This makes it more straightforward to create an alignment database directly from the flattened RODA downloads

- much faster due to the use of threading and mp
- also supports sharding

Bugfix: Corrected paths for just-in-time embedding generation

  instead of pip
- Adds helper line to automatically prepend conda library to
  $LD_LIBRARY_PATH

- much faster due to the use of threading and mp
- also supports sharding

tests in test_data_transforms.py

Previously, long sequences were not excluded from the script.
This commit changes the comparison to exclude sequences with length
greater than args.max_seqlen.