crested.tl.zoo.enformer

Contents

crested.tl.zoo.enformer#

crested.tl.zoo.enformer(seq_len, num_classes, num_conv_blocks=6, num_transformer_blocks=11, num_transformer_heads=8, target_length=896, start_filters=768, filters=1536, pointwise_filters=3072, conv_activation='gelu_enf', transformer_activation='relu', output_activation='softplus', pool_type='attention', first_kernel_size=15, kernel_size=5, transformer_dropout=0.4, pointwise_dropout=0.05, bn_sync=False, name='Enformer')#

Construct an fully replicated Enformer model.

Note that unlike other CREsted model zoo architectures, this architecture is not suited for predicting individual regions out of the box.

Parameters:

  • seq_len (int) – Width of the input region. Enformer default is 196608

  • num_classes (int | Sequence[int]) – Number of classes to predict. If an int, creates a single head with num_classes classes. If a list of integers, creates multiple heads in a list.

  • num_conv_blocks (int (default: 6)) – Number of convolution blocks to include in the tower, after the stem block.

  • num_transformer_blocks (int (default: 11)) – Number of transformer blocks to include in the transformer stack.

  • target_length (int (default: 896)) – The target length in bins to crop to. Default is 896, cropping away 320 bins (41kb) on each side.

  • start_filters (int (default: 768)) – Starting number of filters for the first DNA-facing and first conv tower block, exponentially increasing towards filters through the conv tower.

  • filters (int (default: 1536)) – Number of filters at the end of the conv tower.

  • pointwise_filters (int (default: 3072)) – Number of filters of the post-transformer final pointwise convolution.

  • conv_activation (str (default: 'gelu_enf')) – Activation function to use in the conv tower and in the final pointwise block.

  • transformer_activation (str (default: 'relu')) – Activation function to use in the feedforward section of the transformer blocks.

  • output_activation (str (default: 'softplus')) – Final activation to use on the output heads, just before predicting the tracks.

  • pool_type (str (default: 'attention')) – Pooling type to use, one of ‘max’ or ‘attention’.

  • first_kernel_size (int (default: 15)) – Kernel size of the first conv layer, directly interfacing the sequence.

  • kernel_size (int (default: 5)) – Kernel size of the convolutions in the conv tower.

  • transformer_dropout (default: 0.4) – Dropout rate used in the transformer blocks, both MHA and feed-forward.

  • pointwise_dropout (float (default: 0.05)) – Dropout rate of the post-transformer final pointwise layer.

  • bn_sync (bool (default: False)) – Whether to use synchronized cross-GPU BatchNormalisations. Default is False.

Return type:

Model

Returns:

A Keras model.

Contents