allows one to shortcut sampling at a specific unet number, if one were to be training in stages

README.md

@@ -10,7 +10,7 @@ The main novelty seems to be an extra layer of indirection with the prior networ
 
 This model is SOTA for text-to-image for now.
 
-Please join <a href="https://discord.gg/xBPBXfcFHd"><img alt="Join us on Discord" src="https://img.shields.io/discord/823813159592001537?color=5865F2&logo=discord&logoColor=white"></a> if you are interested in helping out with the replication with the <a href="https://laion.ai/">LAION</a> community | <a href="https://www.youtube.com/watch?v=AIOE1l1W0Tw">Yannic Interview</a>
+Please join <a href="https://discord.gg/xBPBXfcFHd"><img alt="Join us on Discord" src="https://img.shields.io/discord/823813159592001537?color=5865F2&logo=discord&logoColor=white"></a> if you are interested in helping out with the replication
 
 There was enough interest for a <a href="https://github.com/lucidrains/dalle2-jax">Jax version</a>. I will also eventually extend this to <a href="https://github.com/lucidrains/dalle2-video">text to video</a>, once the repository is in a good place.
 
@@ -783,184 +783,9 @@ for unet_number in (1, 2):
 # you can sample from the exponentially moving averaged unets as so
 
 mock_image_embed = torch.randn(4, 512).cuda()
-images = decoder_trainer.sample(mock_image_embed, text = text) # (4, 3, 256, 256)
+images = decoder.sample(mock_image_embed, text = text) # (4, 3, 256, 256)
 ```
 
-### Diffusion Prior Training
-
-Similarly, one can use the `DiffusionPriorTrainer` to automatically instantiate and keep track of an exponential moving averaged prior.
-
-```python
-import torch
-from dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, DiffusionPriorTrainer, Unet, Decoder, CLIP
-
-clip = CLIP(
-    dim_text = 512,
-    dim_image = 512,
-    dim_latent = 512,
-    num_text_tokens = 49408,
-    text_enc_depth = 6,
-    text_seq_len = 256,
-    text_heads = 8,
-    visual_enc_depth = 6,
-    visual_image_size = 256,
-    visual_patch_size = 32,
-    visual_heads = 8
-).cuda()
-
-# mock data
-
-text = torch.randint(0, 49408, (4, 256)).cuda()
-images = torch.randn(4, 3, 256, 256).cuda()
-
-# prior networks (with transformer)
-
-prior_network = DiffusionPriorNetwork(
-    dim = 512,
-    depth = 6,
-    dim_head = 64,
-    heads = 8
-).cuda()
-
-diffusion_prior = DiffusionPrior(
-    net = prior_network,
-    clip = clip,
-    timesteps = 100,
-    cond_drop_prob = 0.2
-).cuda()
-
-diffusion_prior_trainer = DiffusionPriorTrainer(
-    diffusion_prior,
-    lr = 3e-4,
-    wd = 1e-2,
-    ema_beta = 0.99,
-    ema_update_after_step = 1000,
-    ema_update_every = 10,
-)
-
-loss = diffusion_prior_trainer(text, images)
-loss.backward()
-diffusion_prior_trainer.update()  # this will update the optimizer as well as the exponential moving averaged diffusion prior
-
-# after much of the above three lines in a loop
-# you can sample from the exponential moving average of the diffusion prior identically to how you do so for DiffusionPrior
-
-image_embeds = diffusion_prior_trainer.sample(text) # (4, 512) - exponential moving averaged image embeddings
-```
-
-### Decoder Dataloaders
-
-In order to make loading data simple and efficient, we include some general dataloaders that can be used to train portions of the network.
-
-#### Decoder: Image Embedding Dataset
-
-When training the decoder (and up samplers if training together) in isolation, you will need to load images and corresponding image embeddings. This dataset can read two similar types of datasets. First, it can read a [webdataset](https://github.com/webdataset/webdataset) that contains `.jpg` and `.npy` files in the `.tar`s that contain the images and associated image embeddings respectively. Alternatively, you can also specify a source for the embeddings outside of the webdataset. In this case, the path to the embeddings should contain `.npy` files with the same shard numbers as the webdataset and there should be a correspondence between the filename of the `.jpg` and the index of the embedding in the `.npy`. So, for example, `0001.tar` from the webdataset with image `00010509.jpg` (the first 4 digits are the shard number and the last 4 are the index) in it should be paralleled by a `img_emb_0001.npy` which contains a NumPy array with the embedding at index 509.
-
-Generating a dataset of this type: 
-1. Use [img2dataset](https://github.com/rom1504/img2dataset) to generate a webdataset.
-2. Use [clip-retrieval](https://github.com/rom1504/clip-retrieval) to convert the images to embeddings.
-3. Use [embedding-dataset-reordering](https://github.com/Veldrovive/embedding-dataset-reordering) to reorder the embeddings into the expected format.
-
-Usage:
-
-```python
-from dalle2_pytorch.dataloaders import ImageEmbeddingDataset, create_image_embedding_dataloader
-
-# Create a dataloader directly.
-dataloader = create_image_embedding_dataloader(
-    tar_url="/path/or/url/to/webdataset/{0000..9999}.tar", # Uses braket expanding notation. This specifies to read all tars from 0000.tar to 9999.tar
-    embeddings_url="path/or/url/to/embeddings/folder",     # Included if .npy files are not in webdataset. Left out or set to None otherwise
-    num_workers=4,
-    batch_size=32,
-    shard_width=4,                                         # If a file in the webdataset shard 3 is named 0003039.jpg, we know the shard width is 4 and the last three digits are the index
-    shuffle_num=200,                                       # Does a shuffle of the data with a buffer size of 200
-    shuffle_shards=True,                                   # Shuffle the order the shards are read in
-    resample_shards=False,                                 # Sample shards with replacement. If true, an epoch will be infinite unless stopped manually
-)
-for img, emb in dataloader:
-    print(img.shape)  # torch.Size([32, 3, 256, 256])
-    print(emb.shape)  # torch.Size([32, 512])
-    # Train decoder only as shown above
-
-# Or create a dataset without a loader so you can configure it manually
-dataset = ImageEmbeddingDataset(
-    urls="/path/or/url/to/webdataset/{0000..9999}.tar",
-    embedding_folder_url="path/or/url/to/embeddings/folder",
-    shard_width=4,
-    shuffle_shards=True,
-    resample=False
-)
-```
-
-## Scripts
-
-### Using the `train_diffusion_prior.py` script
-
-This script allows training the DiffusionPrior on pre-computed text and image embeddings. The working example below elucidates this process.
-Please note that the script internally passes text_embed and image_embed to the DiffusionPrior, unlike the example below.
-
-### Usage 
-
-```bash
-$ python train_diffusion_prior.py
-```
-
-The most significant parameters for the script are as follows:
-
---image-embed-url, default = "https://mystic.the-eye.eu/public/AI/cah/laion5b/embeddings/laion2B-en/img_emb/")
-
---text-embed-url, default = "https://mystic.the-eye.eu/public/AI/cah/laion5b/embeddings/laion2B-en/text_emb/")
-
---image-embed-dim, default=768 - 768 corresponds to the ViT iL/14 embedding size,change it to what your chosen ViT generates
-
---learning-rate, default=1.1e-4
-
---weight-decay,  default=6.02e-2
-
---max-grad-norm, default=0.5
-
---batch-size, default=10 ** 4
-
---num-epochs, default=5
-
---clip, default=None # Signals the prior to use pre-computed embeddings
-
-### Sample wandb run log
-
-Please find a sample wandb run log at : https://wandb.ai/laion/diffusion-prior/runs/1blxu24j
-
-### Loading and saving the Diffusion Prior model
-
-Two methods are provided, load_diffusion_model and save_diffusion_model, the names being self-explanatory. 
-
-## from dalle2_pytorch import load_diffusion_model, save_diffusion_model
-
-    load_diffusion_model(dprior_path, device) 
-
-        dprior_path : path to saved model(.pth)
-    
-        device      : the cuda device you're running on
-    
-    save_diffusion_model(save_path, model, optimizer, scaler, config, image_embed_dim)
-    
-        save_path : path to save at
-    
-        model     : object of Diffusion_Prior
-    
-        optimizer : optimizer object - see train_diffusion_prior.py for how to create one. 
-    
-            e.g: optimizer = get_optimizer(diffusion_prior.net.parameters(), wd=weight_decay, lr=learning_rate)
-    
-        scaler    : a GradScaler object.
-    
-            e.g: scaler = GradScaler(enabled=amp)
-    
-        config    : config object created in train_diffusion_prior.py - see file for example. 
-    
-        image_embed_dim - the dimension of the image_embedding
-    
-            e.g: 768
-
 ## CLI (wip)
 
 ```bash
@@ -993,28 +818,14 @@ Once built, images will be saved to the same directory the command is invoked
 - [x] bring in vit-vqgan https://arxiv.org/abs/2110.04627 for the latent diffusion
 - [x] abstract interface for CLIP adapter class, so other CLIPs can be brought in
 - [x] take care of mixed precision as well as gradient accumulation within decoder trainer
-- [x] just take care of the training for the decoder in a wrapper class, as each unet in the cascade will need its own optimizer
-- [x] bring in tools to train vqgan-vae
-- [x] add convnext backbone for vqgan-vae (in addition to vit [vit-vqgan] + resnet)
-- [x] make sure DDPMs can be run with traditional resnet blocks (but leave convnext as an option for experimentation)
-- [x] make sure for the latter unets in the cascade, one can train on crops for learning super resolution (constrain the unet to be only convolutions in that case, or allow conv-like attention with rel pos bias)
-- [x] offer setting in diffusion prior to split time and image embeddings into multiple tokens, configurable, for more surface area during attention
-- [ ] become an expert with unets, cleanup unet code, make it fully configurable, port all learnings over to https://github.com/lucidrains/x-unet (test out unet² in ddpm repo) - consider https://github.com/lucidrains/uformer-pytorch attention-based unet
-- [ ] make sure the cascading ddpm in the repository can be trained unconditionally, offer a one-line CLI tool for training on a folder of images
+- [ ] become an expert with unets, cleanup unet code, make it fully configurable, port all learnings over to https://github.com/lucidrains/x-unet
+- [ ] copy the cascading ddpm code to a separate repo (perhaps https://github.com/lucidrains/denoising-diffusion-pytorch) as the main contribution of dalle2 really is just the prior network
 - [ ] transcribe code to Jax, which lowers the activation energy for distributed training, given access to TPUs
-- [ ] pull logic for training diffusion prior into a class DiffusionPriorTrainer, for eventual script based + CLI based training
+- [ ] just take care of the training for the decoder in a wrapper class, as each unet in the cascade will need its own optimizer
 - [ ] train on a toy task, offer in colab
 - [ ] think about how best to design a declarative training config that handles preencoding for prior and training of multiple networks in decoder
 - [ ] extend diffusion head to use diffusion-gan (potentially using lightweight-gan) to speed up inference
-- [ ] bring in cross-scale embedding from iclr paper https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/crossformer.py#L14
-- [ ] figure out if possible to augment with external memory, as described in https://arxiv.org/abs/2204.11824
-- [ ] test out grid attention in cascading ddpm locally, decide whether to keep or remove
-- [ ] use an experimental tracker agnostic setup, as done <a href="https://github.com/lucidrains/tf-bind-transformer#simple-trainer-class-for-fine-tuning">here</a>
-- [ ] interface out the vqgan-vae so a pretrained one can be pulled off the shelf to validate latent diffusion + DALL-E2
-- [ ] make sure FILIP works with DALL-E2 from x-clip https://arxiv.org/abs/2111.07783
-- [ ] make sure resnet hyperparameters can be configurable across unet depth (groups and expansion factor)
-- [ ] offer save / load methods on the trainer classes to automatically take care of state dicts for scalers / optimizers / saving versions and checking for breaking changes
-- [ ] bring in skip-layer excitatons (from lightweight gan paper) to see if it helps for either decoder of unet or vqgan-vae training
+- [ ] bring in tools to train vqgan-vae
 
 ## Citations
 
@@ -1045,12 +856,10 @@ Once built, images will be saved to the same directory the command is invoked
 ```
 
 ```bibtex
-@article{shen2019efficient,
-    author  = {Zhuoran Shen and Mingyuan Zhang and Haiyu Zhao and Shuai Yi and Hongsheng Li},
-    title   = {Efficient Attention: Attention with Linear Complexities},
-    journal = {CoRR},
-    year    = {2018},
-    url     = {http://arxiv.org/abs/1812.01243},
+@inproceedings{Liu2022ACF,
+    title   = {A ConvNet for the 2020https://arxiv.org/abs/2112.11435s},
+    author  = {Zhuang Liu and Hanzi Mao and Chaozheng Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},
+    year    = {2022}
 }
 ```
 
@@ -1072,24 +881,4 @@ Once built, images will be saved to the same directory the command is invoked
 }
 ```
 
-```bibtex
-@article{Shleifer2021NormFormerIT,
-    title   = {NormFormer: Improved Transformer Pretraining with Extra Normalization},
-    author  = {Sam Shleifer and Jason Weston and Myle Ott},
-    journal = {ArXiv},
-    year    = {2021},
-    volume  = {abs/2110.09456}
-}
-```
-
-```bibtex
-@article{Yu2022CoCaCC,
-    title   = {CoCa: Contrastive Captioners are Image-Text Foundation Models},
-    author  = {Jiahui Yu and Zirui Wang and Vijay Vasudevan and Legg Yeung and Mojtaba Seyedhosseini and Yonghui Wu},
-    journal = {ArXiv},
-    year    = {2022},
-    volume  = {abs/2205.01917}
-}
-```
-
-*Creating noise from data is easy; creating data from noise is generative modeling.* - <a href="https://arxiv.org/abs/2011.13456">Yang Song's paper</a>
+*Creating noise from data is easy; creating data from noise is generative modeling.* - Yang Song's <a href="https://arxiv.org/abs/2011.13456">paper</a>

dalle2_pytorch/__init__.py

@@ -1,6 +1,6 @@
-from dalle2_pytorch.dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, Unet, Decoder,load_diffusion_model,save_diffusion_model
+from dalle2_pytorch.dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, Unet, Decoder
 from dalle2_pytorch.dalle2_pytorch import OpenAIClipAdapter
-from dalle2_pytorch.train import DecoderTrainer, DiffusionPriorTrainer
+from dalle2_pytorch.train import DecoderTrainer
 
 from dalle2_pytorch.vqgan_vae import VQGanVAE
 from x_clip import CLIP

dalle2_pytorch/cli.py

@@ -1,7 +1,6 @@
 import click
 import torch
 import torchvision.transforms as T
-from functools import reduce
 from pathlib import Path
 
 from dalle2_pytorch import DALLE2, Decoder, DiffusionPrior

dalle2_pytorch/dalle2_pytorch.py

@@ -4,8 +4,6 @@ from inspect import isfunction
 from functools import partial
 from contextlib import contextmanager
 from collections import namedtuple
-from pathlib import Path
-import time
 
 import torch
 import torch.nn.functional as F
@@ -18,66 +16,19 @@ from einops_exts import rearrange_many, repeat_many, check_shape
 from einops_exts.torch import EinopsToAndFrom
 
 from kornia.filters import gaussian_blur2d
-import kornia.augmentation as K
 
 from dalle2_pytorch.tokenizer import tokenizer
 from dalle2_pytorch.vqgan_vae import NullVQGanVAE, VQGanVAE
 
-from resize_right import resize
-
-# rotary embeddings
-
-from rotary_embedding_torch import RotaryEmbedding
-
 # use x-clip
 
 from x_clip import CLIP
-from coca_pytorch import CoCa
-
-# Diffusion Prior model loading and saving functions
-
-def load_diffusion_model(dprior_path, device ):
-
-        dprior_path = Path(dprior_path)
-        assert dprior_path.exists(), 'Dprior model file does not exist'
-        loaded_obj = torch.load(str(dprior_path), map_location='cpu')
-
-        # Get hyperparameters of loaded model 
-        dpn_config = loaded_obj['hparams']['diffusion_prior_network']
-        dp_config = loaded_obj['hparams']['diffusion_prior']
-        image_embed_dim = loaded_obj['image_embed_dim']['image_embed_dim']
-
-        # Create DiffusionPriorNetwork and DiffusionPrior with loaded hyperparameters
-
-        # DiffusionPriorNetwork 
-        prior_network = DiffusionPriorNetwork( dim = image_embed_dim, **dpn_config).to(device)
-
-        # DiffusionPrior with text embeddings and image embeddings pre-computed
-        diffusion_prior = DiffusionPrior(net = prior_network, **dp_config, image_embed_dim = image_embed_dim).to(device)
-
-        # Load state dict from saved model
-        diffusion_prior.load_state_dict(loaded_obj['model'])
-
-        return diffusion_prior
-
-def save_diffusion_model(save_path, model, optimizer, scaler, config, image_embed_dim):
-    # Saving State Dict
-    print("====================================== Saving checkpoint ======================================")
-    state_dict = dict(model=model.state_dict(), 
-                      optimizer=optimizer.state_dict(), 
-                      scaler=scaler.state_dict(), 
-                      hparams = config, 
-                      image_embed_dim = {"image_embed_dim":image_embed_dim})
-    torch.save(state_dict, save_path+'/'+str(time.time())+'_saved_model.pth')
 
 # helper functions
 
 def exists(val):
     return val is not None
 
-def identity(t, *args, **kwargs):
-    return t
-
 def default(val, d):
     if exists(val):
         return val
@@ -131,14 +82,14 @@ def freeze_model_and_make_eval_(model):
 def l2norm(t):
     return F.normalize(t, dim = -1)
 
-def resize_image_to(image, target_image_size):
-    orig_image_size = image.shape[-1]
+def resize_image_to(t, image_size, mode = 'bilinear'): # take a look at https://github.com/assafshocher/ResizeRight
+    shape = cast_tuple(image_size, 2)
+    orig_image_size = t.shape[-2:]
 
-    if orig_image_size == target_image_size:
-        return image
+    if orig_image_size == shape:
+        return t
 
-    scale_factors = target_image_size / orig_image_size
-    return resize(image, scale_factors = scale_factors)
+    return F.interpolate(t, size = shape, mode = mode, align_corners = False)
 
 # image normalization functions
 # ddpms expect images to be in the range of -1 to 1
@@ -156,10 +107,9 @@ EmbeddedText = namedtuple('EmbedTextReturn', ['text_embed', 'text_encodings', 't
 EmbeddedImage = namedtuple('EmbedImageReturn', ['image_embed', 'image_encodings'])
 
 class BaseClipAdapter(nn.Module):
-    def __init__(self, clip, **kwargs):
+    def __init__(self, clip):
         super().__init__()
         self.clip = clip
-        self.overrides = kwargs
 
     @property
     def dim_latent(self):
@@ -217,39 +167,6 @@ class XClipAdapter(BaseClipAdapter):
         image_embed = self.clip.to_visual_latent(image_cls)
         return EmbeddedImage(l2norm(image_embed), image_encodings)
 
-class CoCaAdapter(BaseClipAdapter):
-    @property
-    def dim_latent(self):
-        return self.clip.dim
-
-    @property
-    def image_size(self):
-        assert 'image_size' in self.overrides
-        return self.overrides['image_size']
-
-    @property
-    def image_channels(self):
-        assert 'image_channels' in self.overrides
-        return self.overrides['image_channels']
-
-    @property
-    def max_text_len(self):
-        assert 'max_text_len' in self.overrides
-        return self.overrides['max_text_len']
-
-    @torch.no_grad()
-    def embed_text(self, text):
-        text = text[..., :self.max_text_len]
-        text_mask = text != 0
-        text_embed, text_encodings = self.clip.embed_text(text)
-        return EmbeddedText(text_embed, text_encodings, text_mask)
-
-    @torch.no_grad()
-    def embed_image(self, image):
-        image = resize_image_to(image, self.image_size)
-        image_embed, image_encodings = self.clip.embed_image(image)
-        return EmbeddedImage(image_embed, image_encodings)
-
 class OpenAIClipAdapter(BaseClipAdapter):
     def __init__(
         self,
@@ -302,7 +219,7 @@ class OpenAIClipAdapter(BaseClipAdapter):
         text_embed = self.clip.encode_text(text)
         text_encodings = self.text_encodings
         del self.text_encodings
-        return EmbeddedText(l2norm(text_embed.float()), text_encodings.float(), text_mask)
+        return EmbeddedText(text_embed.float(), text_encodings.float(), text_mask)
 
     @torch.no_grad()
     def embed_image(self, image):
@@ -310,7 +227,7 @@ class OpenAIClipAdapter(BaseClipAdapter):
         image = resize_image_to(image, self.image_size)
         image = self.clip_normalize(unnormalize_img(image))
         image_embed = self.clip.encode_image(image)
-        return EmbeddedImage(l2norm(image_embed.float()), None)
+        return EmbeddedImage(image_embed.float(), None)
 
 # classifier free guidance functions
 
@@ -579,12 +496,7 @@ class SwiGLU(nn.Module):
         x, gate = x.chunk(2, dim = -1)
         return x * F.silu(gate)
 
-def FeedForward(
-    dim,
-    mult = 4,
-    dropout = 0.,
-    post_activation_norm = False
-):
+def FeedForward(dim, mult = 4, dropout = 0., post_activation_norm = False):
     """ post-activation norm https://arxiv.org/abs/2110.09456 """
 
     inner_dim = int(mult * dim)
@@ -607,9 +519,7 @@ class Attention(nn.Module):
         dim_head = 64,
         heads = 8,
         dropout = 0.,
-        causal = False,
-        post_norm = False,
-        rotary_emb = None
+        causal = False
     ):
         super().__init__()
         self.scale = dim_head ** -0.5
@@ -624,13 +534,7 @@ class Attention(nn.Module):
         self.null_kv = nn.Parameter(torch.randn(2, dim_head))
         self.to_q = nn.Linear(dim, inner_dim, bias = False)
         self.to_kv = nn.Linear(dim, dim_head * 2, bias = False)
-
-        self.rotary_emb = rotary_emb
-
-        self.to_out = nn.Sequential(
-            nn.Linear(inner_dim, dim, bias = False),
-            LayerNorm(dim) if post_norm else nn.Identity()
-        )
+        self.to_out = nn.Linear(inner_dim, dim, bias = False)
 
     def forward(self, x, mask = None, attn_bias = None):
         b, n, device = *x.shape[:2], x.device
@@ -639,12 +543,6 @@ class Attention(nn.Module):
         q, k, v = (self.to_q(x), *self.to_kv(x).chunk(2, dim = -1))
 
         q = rearrange(q, 'b n (h d) -> b h n d', h = self.heads)
-        q = q * self.scale
-
-        # rotary embeddings
-
-        if exists(self.rotary_emb):
-            q, k = map(self.rotary_emb.rotate_queries_or_keys, (q, k))
 
         # add null key / value for classifier free guidance in prior net
 
@@ -652,7 +550,7 @@ class Attention(nn.Module):
         k = torch.cat((nk, k), dim = -2)
         v = torch.cat((nv, v), dim = -2)
 
-        # calculate query / key similarities
+        q = q * self.scale
 
         sim = einsum('b h i d, b j d -> b h i j', q, k)
 
@@ -677,7 +575,7 @@ class Attention(nn.Module):
 
         # attention
 
-        sim = sim - sim.amax(dim = -1, keepdim = True).detach()
+        sim = sim - sim.amax(dim = -1, keepdim = True)
         attn = sim.softmax(dim = -1)
         attn = self.dropout(attn)
 
@@ -698,23 +596,19 @@ class CausalTransformer(nn.Module):
         dim_head = 64,
         heads = 8,
         ff_mult = 4,
-        norm_out = True,
+        norm_out = False,
         attn_dropout = 0.,
         ff_dropout = 0.,
-        final_proj = True,
-        normformer = False,
-        rotary_emb = True
+        final_proj = True
     ):
         super().__init__()
         self.rel_pos_bias = RelPosBias(heads = heads)
 
-        rotary_emb = RotaryEmbedding(dim = min(32, dim_head)) if rotary_emb else None
-
         self.layers = nn.ModuleList([])
         for _ in range(depth):
             self.layers.append(nn.ModuleList([
-                Attention(dim = dim, causal = True, dim_head = dim_head, heads = heads, dropout = attn_dropout, post_norm = normformer, rotary_emb = rotary_emb),
-                FeedForward(dim = dim, mult = ff_mult, dropout = ff_dropout, post_activation_norm = normformer)
+                Attention(dim = dim, causal = True, dim_head = dim_head, heads = heads, dropout = attn_dropout),
+                FeedForward(dim = dim, mult = ff_mult, dropout = ff_dropout)
             ]))
 
         self.norm = LayerNorm(dim) if norm_out else nn.Identity()  # unclear in paper whether they projected after the classic layer norm for the final denoised image embedding, or just had the transformer output it directly: plan on offering both options
@@ -741,31 +635,10 @@ class DiffusionPriorNetwork(nn.Module):
         self,
         dim,
         num_timesteps = None,
-        num_time_embeds = 1,
-        num_image_embeds = 1,
-        num_text_embeds = 1,
         **kwargs
     ):
         super().__init__()
-        self.num_time_embeds = num_time_embeds
-        self.num_image_embeds = num_image_embeds
-        self.num_text_embeds = num_text_embeds
-
-        self.to_text_embeds = nn.Sequential(
-            nn.Linear(dim, dim * num_text_embeds) if num_text_embeds > 1 else nn.Identity(),
-            Rearrange('b (n d) -> b n d', n = num_text_embeds)
-        )
-
-        self.to_time_embeds = nn.Sequential(
-            nn.Embedding(num_timesteps, dim * num_time_embeds) if exists(num_timesteps) else nn.Sequential(SinusoidalPosEmb(dim), MLP(dim, dim * num_time_embeds)), # also offer a continuous version of timestep embeddings, with a 2 layer MLP
-            Rearrange('b (n d) -> b n d', n = num_time_embeds)
-        )
-
-        self.to_image_embeds = nn.Sequential(
-            nn.Linear(dim, dim * num_image_embeds) if num_image_embeds > 1 else nn.Identity(),
-            Rearrange('b (n d) -> b n d', n = num_image_embeds)
-        )
-
+        self.time_embeddings = nn.Embedding(num_timesteps, dim) if exists(num_timesteps) else nn.Sequential(Rearrange('b -> b 1'), MLP(1, dim)) # also offer a continuous version of timestep embeddings, with a 2 layer MLP
         self.learned_query = nn.Parameter(torch.randn(dim))
         self.causal_transformer = CausalTransformer(dim = dim, **kwargs)
 
@@ -795,13 +668,10 @@ class DiffusionPriorNetwork(nn.Module):
     ):
         batch, dim, device, dtype = *image_embed.shape, image_embed.device, image_embed.dtype
 
-        num_time_embeds, num_image_embeds, num_text_embeds = self.num_time_embeds, self.num_image_embeds, self.num_text_embeds
-
         # in section 2.2, last paragraph
         # "... consisting of encoded text, CLIP text embedding, diffusion timestep embedding, noised CLIP image embedding, final embedding for prediction"
 
-        text_embed = self.to_text_embeds(text_embed)
-        image_embed = self.to_image_embeds(image_embed)
+        text_embed, image_embed = rearrange_many((text_embed, image_embed), 'b d -> b 1 d')
 
         # make text encodings optional
         # although the paper seems to suggest it is present <--
@@ -821,17 +691,16 @@ class DiffusionPriorNetwork(nn.Module):
 
         # whether text embedding is masked or not depends on the classifier free guidance conditional masking
 
-        keep_mask = repeat(keep_mask, 'b 1 -> b n', n = num_text_embeds)
         mask = torch.cat((mask, keep_mask), dim = 1)
 
         # whether text embedding is used for conditioning depends on whether text encodings are available for attention (for classifier free guidance, even though it seems from the paper it was not used in the prior ddpm, as the objective is different)
         # but let's just do it right
 
         if exists(mask):
-            attend_padding = 1 + num_time_embeds + num_image_embeds # 1 for learned queries + number of image embeds + time embeds
-            mask = F.pad(mask, (0, attend_padding), value = True) # extend mask for text embedding, noised image embedding, time step embedding, and learned query
+            mask = F.pad(mask, (0, 2), value = True) # extend mask for text embedding, noised image embedding, time step embedding, and learned query
 
-        time_embed = self.to_time_embeds(diffusion_timesteps)
+        time_embed = self.time_embeddings(diffusion_timesteps)
+        time_embed = rearrange(time_embed, 'b d -> b 1 d')
 
         learned_queries = repeat(self.learned_query, 'd -> b 1 d', b = batch)
 
@@ -839,7 +708,6 @@ class DiffusionPriorNetwork(nn.Module):
             text_encodings,
             text_embed,
             time_embed,
-            image_embed,
             learned_queries
         ), dim = -2)
 
@@ -863,16 +731,12 @@ class DiffusionPrior(BaseGaussianDiffusion):
         image_size = None,
         image_channels = 3,
         timesteps = 1000,
-        cond_drop_prob = 0.,
+        cond_drop_prob = 0.2,
         loss_type = "l1",
         predict_x_start = True,
         beta_schedule = "cosine",
         condition_on_text_encodings = True, # the paper suggests this is needed, but you can turn it off for your CLIP preprocessed text embed -> image embed training
-        sampling_clamp_l2norm = False,
-        training_clamp_l2norm = False,
-        init_image_embed_l2norm = False,
-        image_embed_scale = None,           # this is for scaling the l2-normed image embedding, so it is more suitable for gaussian diffusion, as outlined by Katherine (@crowsonkb) https://github.com/lucidrains/DALLE2-pytorch/issues/60#issue-1226116132
-        clip_adapter_overrides = dict()
+        sampling_clamp_l2norm = False
     ):
         super().__init__(
             beta_schedule = beta_schedule,
@@ -882,9 +746,7 @@ class DiffusionPrior(BaseGaussianDiffusion):
 
         if exists(clip):
             if isinstance(clip, CLIP):
-                clip = XClipAdapter(clip, **clip_adapter_overrides)
-            elif isinstance(clip, CoCa):
-                clip = CoCaAdapter(clip, **clip_adapter_overrides)
+                clip = XClipAdapter(clip)
 
             assert isinstance(clip, BaseClipAdapter)
             freeze_model_and_make_eval_(clip)
@@ -900,16 +762,11 @@ class DiffusionPrior(BaseGaussianDiffusion):
         self.cond_drop_prob = cond_drop_prob
         self.condition_on_text_encodings = condition_on_text_encodings
 
-        # in paper, they do not predict the noise, but predict x0 directly for image embedding, claiming empirically better results. I'll just offer both.
         self.predict_x_start = predict_x_start
-
-        # @crowsonkb 's suggestion - https://github.com/lucidrains/DALLE2-pytorch/issues/60#issue-1226116132
-        self.image_embed_scale = default(image_embed_scale, self.image_embed_dim ** 0.5)
+        # in paper, they do not predict the noise, but predict x0 directly for image embedding, claiming empirically better results. I'll just offer both.
 
         # whether to force an l2norm, similar to clipping denoised, when sampling
         self.sampling_clamp_l2norm = sampling_clamp_l2norm
-        self.training_clamp_l2norm = training_clamp_l2norm
-        self.init_image_embed_l2norm = init_image_embed_l2norm
 
     def p_mean_variance(self, x, t, text_cond, clip_denoised: bool):
         pred = self.net(x, t, **text_cond)
@@ -925,12 +782,12 @@ class DiffusionPrior(BaseGaussianDiffusion):
             x_recon.clamp_(-1., 1.)
 
         if self.predict_x_start and self.sampling_clamp_l2norm:
-            x_recon = l2norm(x_recon) * self.image_embed_scale
+            x_recon = l2norm(x_recon)
 
         model_mean, posterior_variance, posterior_log_variance = self.q_posterior(x_start=x_recon, x_t=x, t=t)
         return model_mean, posterior_variance, posterior_log_variance
 
-    @torch.inference_mode()
+    @torch.no_grad()
     def p_sample(self, x, t, text_cond = None, clip_denoised = True, repeat_noise = False):
         b, *_, device = *x.shape, x.device
         model_mean, _, model_log_variance = self.p_mean_variance(x = x, t = t, text_cond = text_cond, clip_denoised = clip_denoised)
@@ -939,21 +796,16 @@ class DiffusionPrior(BaseGaussianDiffusion):
         nonzero_mask = (1 - (t == 0).float()).reshape(b, *((1,) * (len(x.shape) - 1)))
         return model_mean + nonzero_mask * (0.5 * model_log_variance).exp() * noise
 
-    @torch.inference_mode()
+    @torch.no_grad()
     def p_sample_loop(self, shape, text_cond):
         device = self.betas.device
 
         b = shape[0]
-        image_embed = torch.randn(shape, device=device)
-
-        if self.init_image_embed_l2norm:
-            image_embed = l2norm(image_embed) * self.image_embed_scale
+        img = torch.randn(shape, device=device)
 
         for i in tqdm(reversed(range(0, self.num_timesteps)), desc='sampling loop time step', total=self.num_timesteps):
-            times = torch.full((b,), i, device = device, dtype = torch.long)
-            image_embed = self.p_sample(image_embed, times, text_cond = text_cond)
-
-        return image_embed
+            img = self.p_sample(img, torch.full((b,), i, device = device, dtype = torch.long), text_cond = text_cond)
+        return img
 
     def p_losses(self, image_embed, times, text_cond, noise = None):
         noise = default(noise, lambda: torch.randn_like(image_embed))
@@ -967,27 +819,12 @@ class DiffusionPrior(BaseGaussianDiffusion):
             **text_cond
         )
 
-        if self.predict_x_start and self.training_clamp_l2norm:
-            pred = l2norm(pred) * self.image_embed_scale
-
         target = noise if not self.predict_x_start else image_embed
 
         loss = self.loss_fn(pred, target)
         return loss
 
-    @torch.inference_mode()
-    @eval_decorator
-    def sample_batch_size(self, batch_size, text_cond):
-        device = self.betas.device
-        shape = (batch_size, self.image_embed_dim)
-
-        img = torch.randn(shape, device = device)
-
-        for i in tqdm(reversed(range(0, self.num_timesteps)), desc = 'sampling loop time step', total = self.num_timesteps):
-            img = self.p_sample(img, torch.full((batch_size,), i, device = device, dtype = torch.long), text_cond = text_cond)
-        return img
-
-    @torch.inference_mode()
+    @torch.no_grad()
     @eval_decorator
     def sample(self, text, num_samples_per_batch = 2):
         # in the paper, what they did was
@@ -1005,11 +842,6 @@ class DiffusionPrior(BaseGaussianDiffusion):
             text_cond = {**text_cond, 'text_encodings': text_encodings, 'mask': text_mask}
 
         image_embeds = self.p_sample_loop((batch_size, image_embed_dim), text_cond = text_cond)
-
-        # retrieve original unscaled image embed
-
-        image_embeds /= self.image_embed_scale
-
         text_embeds = text_cond['text_embed']
 
         text_embeds = rearrange(text_embeds, '(b r) d -> b r d', r = num_samples_per_batch)
@@ -1057,10 +889,6 @@ class DiffusionPrior(BaseGaussianDiffusion):
         batch, device = image_embed.shape[0], image_embed.device
         times = torch.randint(0, self.num_timesteps, (batch,), device = device, dtype = torch.long)
 
-        # scale image embed (Katherine)
-
-        image_embed *= self.image_embed_scale
-
         # calculate forward loss
 
         return self.p_losses(image_embed, times, text_cond = text_cond, *args, **kwargs)
@@ -1085,41 +913,20 @@ class SinusoidalPosEmb(nn.Module):
         emb = rearrange(x, 'i -> i 1') * rearrange(emb, 'j -> 1 j')
         return torch.cat((emb.sin(), emb.cos()), dim = -1)
 
-class Block(nn.Module):
-    def __init__(
-        self,
-        dim,
-        dim_out,
-        groups = 8
-    ):
-        super().__init__()
-        self.block = nn.Sequential(
-            nn.Conv2d(dim, dim_out, 3, padding = 1),
-            nn.GroupNorm(groups, dim_out),
-            nn.SiLU()
-        )
-    def forward(self, x):
-        return self.block(x)
+class ConvNextBlock(nn.Module):
+    """ https://arxiv.org/abs/2201.03545 """
 
-class ResnetBlock(nn.Module):
     def __init__(
         self,
         dim,
         dim_out,
         *,
         cond_dim = None,
-        time_cond_dim = None,
-        groups = 8
+        mult = 2,
+        norm = True
     ):
         super().__init__()
-
-        self.time_mlp = None
-
-        if exists(time_cond_dim):
-            self.time_mlp = nn.Sequential(
-                nn.SiLU(),
-                nn.Linear(time_cond_dim, dim_out)
-            )
+        need_projection = dim != dim_out
 
         self.cross_attn = None
 
@@ -1128,27 +935,32 @@ class ResnetBlock(nn.Module):
                 'b c h w',
                 'b (h w) c',
                 CrossAttention(
-                    dim = dim_out,
+                    dim = dim,
                     context_dim = cond_dim
                 )
             )
 
-        self.block1 = Block(dim, dim_out, groups = groups)
-        self.block2 = Block(dim_out, dim_out, groups = groups)
-        self.res_conv = nn.Conv2d(dim, dim_out, 1) if dim != dim_out else nn.Identity()
+        self.ds_conv = nn.Conv2d(dim, dim, 7, padding = 3, groups = dim)
 
-    def forward(self, x, cond = None, time_emb = None):
-        h = self.block1(x)
+        inner_dim = int(dim_out * mult)
+        self.net = nn.Sequential(
+            ChanLayerNorm(dim) if norm else nn.Identity(),
+            nn.Conv2d(dim, inner_dim, 3, padding = 1),
+            nn.GELU(),
+            nn.Conv2d(inner_dim, dim_out, 3, padding = 1)
+        )
 
-        if exists(self.time_mlp) and exists(time_emb):
-            time_emb = self.time_mlp(time_emb)
-            h = rearrange(time_emb, 'b c -> b c 1 1') + h
+        self.res_conv = nn.Conv2d(dim, dim_out, 1) if need_projection else nn.Identity()
+
+    def forward(self, x, cond = None):
+        h = self.ds_conv(x)
 
         if exists(self.cross_attn):
             assert exists(cond)
             h = self.cross_attn(h, context = cond) + h
 
-        h = self.block2(h)
+        h = self.net(h)
+
         return h + self.res_conv(x)
 
 class CrossAttention(nn.Module):
@@ -1204,7 +1016,7 @@ class CrossAttention(nn.Module):
             mask = rearrange(mask, 'b j -> b 1 1 j')
             sim = sim.masked_fill(~mask, max_neg_value)
 
-        sim = sim - sim.amax(dim = -1, keepdim = True).detach()
+        sim = sim - sim.amax(dim = -1, keepdim = True)
         attn = sim.softmax(dim = -1)
 
         out = einsum('b h i j, b h j d -> b h i d', attn, v)
@@ -1225,46 +1037,6 @@ class GridAttention(nn.Module):
         out = rearrange(out, '(b h w) (w1 w2) c -> b c (w1 h) (w2 w)', w1 = wsz, w2 = wsz, h = h // wsz, w = w // wsz)
         return out
 
-class LinearAttention(nn.Module):
-    def __init__(
-        self,
-        dim,
-        dim_head = 32,
-        heads = 8
-    ):
-        super().__init__()
-        self.scale = dim_head ** -0.5
-        self.heads = heads
-        inner_dim = dim_head * heads
-        self.norm = ChanLayerNorm(dim)
-
-        self.nonlin = nn.GELU()
-        self.to_qkv = nn.Conv2d(dim, inner_dim * 3, 1, bias = False)
-
-        self.to_out = nn.Sequential(
-            nn.Conv2d(inner_dim, dim, 1, bias = False),
-            ChanLayerNorm(dim)
-        )
-
-    def forward(self, fmap):
-        h, x, y = self.heads, *fmap.shape[-2:]
-
-        fmap = self.norm(fmap)
-        q, k, v = self.to_qkv(fmap).chunk(3, dim = 1)
-        q, k, v = rearrange_many((q, k, v), 'b (h c) x y -> (b h) (x y) c', h = h)
-
-        q = q.softmax(dim = -1)
-        k = k.softmax(dim = -2)
-
-        q = q * self.scale
-
-        context = einsum('b n d, b n e -> b d e', k, v)
-        out = einsum('b n d, b d e -> b n e', q, context)
-        out = rearrange(out, '(b h) (x y) d -> b (h d) x y', h = h, x = x, y = y)
-
-        out = self.nonlin(out)
-        return self.to_out(out)
-
 class Unet(nn.Module):
     def __init__(
         self,
@@ -1279,18 +1051,14 @@ class Unet(nn.Module):
         dim_mults=(1, 2, 4, 8),
         channels = 3,
         attn_dim_head = 32,
-        attn_heads = 16,
+        attn_heads = 8,
         lowres_cond = False, # for cascading diffusion - https://cascaded-diffusion.github.io/
         sparse_attn = False,
+        sparse_attn_window = 8,  # window size for sparse attention
         attend_at_middle = True, # whether to have a layer of attention at the bottleneck (can turn off for higher resolution in cascading DDPM, before bringing in efficient attention)
         cond_on_text_encodings = False,
         max_text_len = 256,
         cond_on_image_embeds = False,
-        init_dim = None,
-        init_conv_kernel_size = 7,
-        block_type = 'resnet',
-        block_resnet_groups = 8,
-        **kwargs
     ):
         super().__init__()
         # save locals to take care of some hyperparameters for cascading DDPM
@@ -1308,34 +1076,22 @@ class Unet(nn.Module):
         self.channels = channels
 
         init_channels = channels if not lowres_cond else channels * 2 # in cascading diffusion, one concats the low resolution image, blurred, for conditioning the higher resolution synthesis
-        init_dim = default(init_dim, dim // 2)
 
-        assert (init_conv_kernel_size % 2) == 1
-        self.init_conv = nn.Conv2d(init_channels, init_dim, init_conv_kernel_size, padding = init_conv_kernel_size // 2)
-
-        dims = [init_dim, *map(lambda m: dim * m, dim_mults)]
+        dims = [init_channels, *map(lambda m: dim * m, dim_mults)]
         in_out = list(zip(dims[:-1], dims[1:]))
 
         # time, image embeddings, and optional text encoding
 
         cond_dim = default(cond_dim, dim)
-        time_cond_dim = dim * 4
 
-        self.to_time_hiddens = nn.Sequential(
+        self.time_mlp = nn.Sequential(
             SinusoidalPosEmb(dim),
-            nn.Linear(dim, time_cond_dim),
-            nn.GELU()
-        )
-
-        self.to_time_tokens = nn.Sequential(
-            nn.Linear(time_cond_dim, cond_dim * num_time_tokens),
+            nn.Linear(dim, dim * 4),
+            nn.GELU(),
+            nn.Linear(dim * 4, cond_dim * num_time_tokens),
             Rearrange('b (r d) -> b r d', r = num_time_tokens)
         )
 
-        self.to_time_cond = nn.Sequential(
-            nn.Linear(time_cond_dim, time_cond_dim)
-        )
-
         self.image_to_cond = nn.Sequential(
             nn.Linear(image_embed_dim, cond_dim * num_image_tokens),
             Rearrange('b (n d) -> b n d', n = num_image_tokens)
@@ -1365,10 +1121,6 @@ class Unet(nn.Module):
 
         attn_kwargs = dict(heads = attn_heads, dim_head = attn_dim_head)
 
-        # resnet block klass
-
-        block_klass = partial(ResnetBlock, groups = block_resnet_groups)
-
         # layers
 
         self.downs = nn.ModuleList([])
@@ -1381,32 +1133,32 @@ class Unet(nn.Module):
             layer_cond_dim = cond_dim if not is_first else None
 
             self.downs.append(nn.ModuleList([
-                block_klass(dim_in, dim_out, time_cond_dim = time_cond_dim),
-                Residual(LinearAttention(dim_out, **attn_kwargs)) if sparse_attn else nn.Identity(),
-                block_klass(dim_out, dim_out, cond_dim = layer_cond_dim, time_cond_dim = time_cond_dim),
+                ConvNextBlock(dim_in, dim_out, norm = ind != 0),
+                Residual(GridAttention(dim_out, window_size = sparse_attn_window, **attn_kwargs)) if sparse_attn else nn.Identity(),
+                ConvNextBlock(dim_out, dim_out, cond_dim = layer_cond_dim),
                 Downsample(dim_out) if not is_last else nn.Identity()
             ]))
 
         mid_dim = dims[-1]
 
-        self.mid_block1 = block_klass(mid_dim, mid_dim, cond_dim = cond_dim, time_cond_dim = time_cond_dim)
+        self.mid_block1 = ConvNextBlock(mid_dim, mid_dim, cond_dim = cond_dim)
         self.mid_attn = EinopsToAndFrom('b c h w', 'b (h w) c', Residual(Attention(mid_dim, **attn_kwargs))) if attend_at_middle else None
-        self.mid_block2 = block_klass(mid_dim, mid_dim, cond_dim = cond_dim, time_cond_dim = time_cond_dim)
+        self.mid_block2 = ConvNextBlock(mid_dim, mid_dim, cond_dim = cond_dim)
 
         for ind, (dim_in, dim_out) in enumerate(reversed(in_out[1:])):
             is_last = ind >= (num_resolutions - 2)
             layer_cond_dim = cond_dim if not is_last else None
 
             self.ups.append(nn.ModuleList([
-                block_klass(dim_out * 2, dim_in, cond_dim = layer_cond_dim, time_cond_dim = time_cond_dim),
-                Residual(LinearAttention(dim_in, **attn_kwargs)) if sparse_attn else nn.Identity(),
-                block_klass(dim_in, dim_in, cond_dim = layer_cond_dim, time_cond_dim = time_cond_dim),
+                ConvNextBlock(dim_out * 2, dim_in, cond_dim = layer_cond_dim),
+                Residual(GridAttention(dim_in, window_size = sparse_attn_window, **attn_kwargs)) if sparse_attn else nn.Identity(),
+                ConvNextBlock(dim_in, dim_in, cond_dim = layer_cond_dim),
                 Upsample(dim_in)
             ]))
 
         out_dim = default(out_dim, channels)
         self.final_conv = nn.Sequential(
-            block_klass(dim, dim),
+            ConvNextBlock(dim, dim),
             nn.Conv2d(dim, out_dim, 1)
         )
 
@@ -1462,16 +1214,9 @@ class Unet(nn.Module):
         if exists(lowres_cond_img):
             x = torch.cat((x, lowres_cond_img), dim = 1)
 
-        # initial convolution
-
-        x = self.init_conv(x)
-
         # time conditioning
 
-        time_hiddens = self.to_time_hiddens(time)
-
-        time_tokens = self.to_time_tokens(time_hiddens)
-        t = self.to_time_cond(time_hiddens)
+        time_tokens = self.time_mlp(time)
 
         # conditional dropout
 
@@ -1537,25 +1282,25 @@ class Unet(nn.Module):
 
         hiddens = []
 
-        for block1, sparse_attn, block2, downsample in self.downs:
-            x = block1(x, c, t)
+        for convnext, sparse_attn, convnext2, downsample in self.downs:
+            x = convnext(x, c)
             x = sparse_attn(x)
-            x = block2(x, c, t)
+            x = convnext2(x, c)
             hiddens.append(x)
             x = downsample(x)
 
-        x = self.mid_block1(x, mid_c, t)
+        x = self.mid_block1(x, mid_c)
 
         if exists(self.mid_attn):
             x = self.mid_attn(x)
 
-        x = self.mid_block2(x, mid_c, t)
+        x = self.mid_block2(x, mid_c)
 
-        for block1, sparse_attn, block2, upsample in self.ups:
+        for convnext, sparse_attn, convnext2, upsample in self.ups:
             x = torch.cat((x, hiddens.pop()), dim=1)
-            x = block1(x, c, t)
+            x = convnext(x, c)
             x = sparse_attn(x)
-            x = block2(x, c, t)
+            x = convnext2(x, c)
             x = upsample(x)
 
         return self.final_conv(x)
@@ -1563,11 +1308,13 @@ class Unet(nn.Module):
 class LowresConditioner(nn.Module):
     def __init__(
         self,
+        cond_upsample_mode = 'bilinear',
         downsample_first = True,
         blur_sigma = 0.1,
         blur_kernel_size = 3,
     ):
         super().__init__()
+        self.cond_upsample_mode = cond_upsample_mode
         self.downsample_first = downsample_first
         self.blur_sigma = blur_sigma
         self.blur_kernel_size = blur_kernel_size
@@ -1581,8 +1328,10 @@ class LowresConditioner(nn.Module):
         blur_sigma = None,
         blur_kernel_size = None
     ):
+        target_image_size = cast_tuple(target_image_size, 2)
+
         if self.training and self.downsample_first and exists(downsample_image_size):
-            cond_fmap = resize_image_to(cond_fmap, downsample_image_size)
+            cond_fmap = resize_image_to(cond_fmap, downsample_image_size, mode = self.cond_upsample_mode)
 
         if self.training:
             # when training, blur the low resolution conditional image
@@ -1590,7 +1339,7 @@ class LowresConditioner(nn.Module):
             blur_kernel_size = default(blur_kernel_size, self.blur_kernel_size)
             cond_fmap = gaussian_blur2d(cond_fmap, cast_tuple(blur_kernel_size, 2), cast_tuple(blur_sigma, 2))
 
-        cond_fmap = resize_image_to(cond_fmap, target_image_size)
+        cond_fmap = resize_image_to(cond_fmap, target_image_size, mode = self.cond_upsample_mode)
 
         return cond_fmap
 
@@ -1599,9 +1348,7 @@ class Decoder(BaseGaussianDiffusion):
         self,
         unet,
         *,
-        clip = None,
-        image_size = None,
-        channels = 3,
+        clip,
         vae = tuple(),
         timesteps = 1000,
         image_cond_drop_prob = 0.1,
@@ -1611,14 +1358,13 @@ class Decoder(BaseGaussianDiffusion):
         predict_x_start = False,
         predict_x_start_for_latent_diffusion = False,
         image_sizes = None,                         # for cascading ddpm, image size at each stage
-        random_crop_sizes = None,                   # whether to random crop the image at that stage in the cascade (super resoluting convolutions at the end may be able to generalize on smaller crops)
+        lowres_cond_upsample_mode = 'bilinear',     # cascading ddpm - low resolution upsample mode
         lowres_downsample_first = True,             # cascading ddpm - resizes to lower resolution, then to next conditional resolution + blur
         blur_sigma = 0.1,                           # cascading ddpm - blur sigma
         blur_kernel_size = 3,                       # cascading ddpm - blur kernel size
         condition_on_text_encodings = False,        # the paper suggested that this didn't do much in the decoder, but i'm allowing the option for experimentation
         clip_denoised = True,
-        clip_x_start = True,
-        clip_adapter_overrides = dict()
+        clip_x_start = True
     ):
         super().__init__(
             beta_schedule = beta_schedule,
@@ -1626,24 +1372,15 @@ class Decoder(BaseGaussianDiffusion):
             loss_type = loss_type
         )
 
-        assert exists(clip) ^ exists(image_size), 'either CLIP is supplied, or you must give the image_size and channels (usually 3 for RGB)'
+        if isinstance(clip, CLIP):
+            clip = XClipAdapter(clip)
 
-        self.clip = None
-        if exists(clip):
-            if isinstance(clip, CLIP):
-                clip = XClipAdapter(clip, **clip_adapter_overrides)
-            elif isinstance(clip, CoCa):
-                clip = CoCaAdapter(clip, **clip_adapter_overrides)
+        freeze_model_and_make_eval_(clip)
+        assert isinstance(clip, BaseClipAdapter)
 
-            freeze_model_and_make_eval_(clip)
-            assert isinstance(clip, BaseClipAdapter)
-
-            self.clip = clip
-            self.clip_image_size = clip.image_size
-            self.channels = clip.image_channels
-        else:
-            self.clip_image_size = image_size
-            self.channels = channels
+        self.clip = clip
+        self.clip_image_size = clip.image_size
+        self.channels = clip.image_channels
 
         self.condition_on_text_encodings = condition_on_text_encodings
 
@@ -1676,17 +1413,13 @@ class Decoder(BaseGaussianDiffusion):
 
         # unet image sizes
 
-        image_sizes = default(image_sizes, (self.clip_image_size,))
+        image_sizes = default(image_sizes, (clip.image_size,))
         image_sizes = tuple(sorted(set(image_sizes)))
 
         assert len(self.unets) == len(image_sizes), f'you did not supply the correct number of u-nets ({len(self.unets)}) for resolutions {image_sizes}'
         self.image_sizes = image_sizes
         self.sample_channels = cast_tuple(self.channels, len(image_sizes))
 
-        # random crop sizes (for super-resoluting unets at the end of cascade?)
-
-        self.random_crop_sizes = cast_tuple(random_crop_sizes, len(image_sizes))
-
         # predict x0 config
 
         self.predict_x_start = cast_tuple(predict_x_start, len(unets)) if not predict_x_start_for_latent_diffusion else tuple(map(lambda t: isinstance(t, VQGanVAE), self.vaes))
@@ -1697,6 +1430,7 @@ class Decoder(BaseGaussianDiffusion):
         assert lowres_conditions == (False, *((True,) * (len(self.unets) - 1))), 'the first unet must be unconditioned (by low resolution image), and the rest of the unets must have `lowres_cond` set to True'
 
         self.to_lowres_cond = LowresConditioner(
+            cond_upsample_mode = lowres_cond_upsample_mode,
             downsample_first = lowres_downsample_first,
             blur_sigma = blur_sigma,
             blur_kernel_size = blur_kernel_size,
@@ -1731,6 +1465,12 @@ class Decoder(BaseGaussianDiffusion):
         yield
         unet.cpu()
 
+
+    @torch.no_grad()
+    def get_image_embed(self, image):
+        image_embed, _ = self.clip.embed_image(image)
+        return image_embed
+
     def p_mean_variance(self, unet, x, t, image_embed, text_encodings = None, text_mask = None, lowres_cond_img = None, clip_denoised = True, predict_x_start = False, cond_scale = 1.):
         pred = unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, text_mask = text_mask, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img)
 
@@ -1745,7 +1485,7 @@ class Decoder(BaseGaussianDiffusion):
         model_mean, posterior_variance, posterior_log_variance = self.q_posterior(x_start=x_recon, x_t=x, t=t)
         return model_mean, posterior_variance, posterior_log_variance
 
-    @torch.inference_mode()
+    @torch.no_grad()
     def p_sample(self, unet, x, t, image_embed, text_encodings = None, text_mask = None, cond_scale = 1., lowres_cond_img = None, predict_x_start = False, clip_denoised = True, repeat_noise = False):
         b, *_, device = *x.shape, x.device
         model_mean, _, model_log_variance = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, text_mask = text_mask, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, clip_denoised = clip_denoised, predict_x_start = predict_x_start)
@@ -1754,7 +1494,7 @@ class Decoder(BaseGaussianDiffusion):
         nonzero_mask = (1 - (t == 0).float()).reshape(b, *((1,) * (len(x.shape) - 1)))
         return model_mean + nonzero_mask * (0.5 * model_log_variance).exp() * noise
 
-    @torch.inference_mode()
+    @torch.no_grad()
     def p_sample_loop(self, unet, shape, image_embed, predict_x_start = False, clip_denoised = True, lowres_cond_img = None, text_encodings = None, text_mask = None, cond_scale = 1):
         device = self.betas.device
 
@@ -1798,7 +1538,7 @@ class Decoder(BaseGaussianDiffusion):
         loss = self.loss_fn(pred, target)
         return loss
 
-    @torch.inference_mode()
+    @torch.no_grad()
     @eval_decorator
     def sample(
         self,
@@ -1869,10 +1609,10 @@ class Decoder(BaseGaussianDiffusion):
 
         unet = self.get_unet(unet_number)
 
-        vae                 = self.vaes[unet_index]
-        target_image_size   = self.image_sizes[unet_index]
-        predict_x_start     = self.predict_x_start[unet_index]
-        random_crop_size    = self.random_crop_sizes[unet_index]
+        target_image_size = self.image_sizes[unet_index]
+        vae = self.vaes[unet_index]
+        predict_x_start = self.predict_x_start[unet_index]
+
         b, c, h, w, device, = *image.shape, image.device
 
         check_shape(image, 'b c h w', c = self.channels)
@@ -1881,12 +1621,10 @@ class Decoder(BaseGaussianDiffusion):
         times = torch.randint(0, self.num_timesteps, (b,), device = device, dtype = torch.long)
 
         if not exists(image_embed):
-            assert exists(self.clip), 'if you want to derive CLIP image embeddings automatically, you must supply `clip` to the decoder on init'
             image_embed, _ = self.clip.embed_image(image)
 
         text_encodings = text_mask = None
         if exists(text) and not exists(text_encodings):
-            assert exists(self.clip), 'if you are passing in raw text, you need to supply `clip` to the decoder'
             _, text_encodings, text_mask = self.clip.embed_text(text)
 
         assert not (self.condition_on_text_encodings and not exists(text_encodings)), 'text or text encodings must be passed into decoder if specified'
@@ -1895,14 +1633,6 @@ class Decoder(BaseGaussianDiffusion):
         lowres_cond_img = self.to_lowres_cond(image, target_image_size = target_image_size, downsample_image_size = self.image_sizes[unet_index - 1]) if unet_number > 1 else None
         image = resize_image_to(image, target_image_size)
 
-        if exists(random_crop_size):
-            aug = K.RandomCrop((random_crop_size, random_crop_size), p = 1.)
-
-            # make sure low res conditioner and image both get augmented the same way
-            # detailed https://kornia.readthedocs.io/en/latest/augmentation.module.html?highlight=randomcrop#kornia.augmentation.RandomCrop
-            image = aug(image)
-            lowres_cond_img = aug(lowres_cond_img, params = aug._params)
-
         vae.eval()
         with torch.no_grad():
             image = vae.encode(image)
@@ -1933,7 +1663,7 @@ class DALLE2(nn.Module):
 
         self.to_pil = T.ToPILImage()
 
-    @torch.inference_mode()
+    @torch.no_grad()
     @eval_decorator
     def forward(
         self,
@@ -1960,4 +1690,3 @@ class DALLE2(nn.Module):
             return images[0]
 
         return images
-

dalle2_pytorch/dataloaders/__init__.py

@@ -1 +0,0 @@
-from dalle2_pytorch.dataloaders.decoder_loader import ImageEmbeddingDataset, create_image_embedding_dataloader

dalle2_pytorch/dataloaders/decoder_loader.py

@@ -1,170 +0,0 @@
-import os
-import webdataset as wds
-import torch
-import numpy as np
-import fsspec
-
-def get_shard(filename):
-    """
-    Filenames with shards in them have a consistent structure that we can take advantage of
-    Standard structure: path/to/file/prefix_string_00001.ext
-    """
-    try:
-        return filename.split("_")[-1].split(".")[0]
-    except ValueError:
-        raise RuntimeError(f"Could not find shard for filename {filename}")
-
-def get_example_file(fs, path, file_format):
-    """
-    Given a file system and a file extension, return the example file
-    """
-    return fs.glob(os.path.join(path, f"*.{file_format}"))[0]
-
-def embedding_inserter(samples, embeddings_url, shard_width, handler=wds.handlers.reraise_exception):
-    """Given a datum of {"__key__": str, "__url__": str, ...} adds the cooresponding embedding and yields"""
-    previous_tar_url = None
-    current_embeddings = None
-    # Get a reference to an abstract file system where the embeddings are stored
-    embeddings_fs, embeddings_path = fsspec.core.url_to_fs(embeddings_url)
-    example_embedding_file = get_example_file(embeddings_fs, embeddings_path, "npy")
-    example_embedding_shard = get_shard(example_embedding_file)
-    emb_shard_width = len(example_embedding_shard)
-    # Easier to get the basename without the shard once than search through for the correct file every time
-    embedding_file_basename = '_'.join(example_embedding_file.split("_")[:-1]) + "_"
-
-    def load_corresponding_embeds(tar_url):
-      """Finds and reads the npy files that contains embeddings for the given webdataset tar"""
-      shard = int(tar_url.split("/")[-1].split(".")[0])
-      embedding_url = embedding_file_basename + str(shard).zfill(emb_shard_width) + '.npy'
-      with embeddings_fs.open(embedding_url) as f:
-        data = np.load(f)
-      return torch.from_numpy(data)
-
-    for sample in samples:
-        try:
-            tar_url = sample["__url__"]
-            key = sample["__key__"]
-            if tar_url != previous_tar_url:
-                # If the tar changed, we need to download new embeddings
-                # This means if we shuffle before inserting it will load many more files than we expect and be very inefficient.
-                previous_tar_url = tar_url
-                current_embeddings = load_corresponding_embeds(tar_url)
-                
-            embedding_index = int(key[shard_width:])
-            sample["npy"] = current_embeddings[embedding_index]
-            yield sample
-        except Exception as exn:  # From wds implementation
-            if handler(exn):
-                continue
-            else:
-                break
-insert_embedding = wds.filters.pipelinefilter(embedding_inserter)
-
-def verify_keys(samples, handler=wds.handlers.reraise_exception):
-    """
-    Requires that both the image and embedding are present in the sample
-    This is important to do as a user may forget they do not have embeddings in their webdataset and neglect to add them using the embedding_folder_url parameter.
-    """
-    for sample in samples:
-        try:
-            assert "jpg" in sample, f"Sample {sample['__key__']} missing image"
-            assert "npy" in sample, f"Sample {sample['__key__']} missing embedding. Did you set embedding_folder_url?"
-            yield sample
-        except Exception as exn:  # From wds implementation
-            if handler(exn):
-                continue
-            else:
-                break
-
-class ImageEmbeddingDataset(wds.DataPipeline, wds.compat.FluidInterface):
-    """
-    A fluid interface wrapper for DataPipline that returns image embedding pairs
-    Reads embeddings as npy files from the webdataset if they exist. If embedding_folder_url is set, they will be inserted in from the alternate source.
-    """
-
-    def __init__(
-            self,
-            urls,
-            embedding_folder_url=None,
-            shard_width=None,
-            handler=wds.handlers.reraise_exception,
-            resample=False,
-            shuffle_shards=True
-    ):
-        """
-        Modeled directly off of the WebDataset constructor
-
-        :param urls: A url pointing to the tar files of the webdataset formatted as /path/to/webdataset/{0000..9999}.tar
-        :param embedding_folder_url: Required if webdataset does not contain embeddings. A url pointing to the npy files of the embeddings. Should have the same number of shards as the webdataset.
-            Webdataset image keys should align with the index of the embedding. This means missing image indices must have a corresponding embedding of all zeros.
-        :param shard_width: The number of digits in the shard number. This is used to align the embedding index with the image index.
-            For example, if a file in the webdataset shard 3 is named 0003039.jpg, we know the shard with this 4 and the last three digits are the index.
-        :param handler: A webdataset handler.
-        :param resample: If true, resample webdataset shards with replacement. You need to set your own epoch size if this is true since it will resample infinitely.
-        :param shuffle_shards: If true, shuffle the shards before resampling. This cannot be true if resample is true.
-        """
-        super().__init__()
-        # Add the shardList and randomize or resample if requested
-        if resample:
-            assert not shuffle_shards, "Cannot both resample and shuffle"
-            self.append(wds.ResampledShards(urls))
-        else:
-            self.append(wds.SimpleShardList(urls))
-            if shuffle_shards:
-                self.append(wds.filters.shuffle(1000))
-
-        self.append(wds.split_by_node)
-        self.append(wds.split_by_worker)
-
-        self.append(wds.tarfile_to_samples(handler=handler))
-        self.append(wds.decode("torchrgb"))
-        if embedding_folder_url is not None:
-            assert shard_width is not None, "Reading embeddings separately requires shard length to be given"
-            self.append(insert_embedding(embeddings_url=embedding_folder_url, shard_width=shard_width, handler=handler))
-        self.append(verify_keys)
-        self.append(wds.to_tuple("jpg", "npy"))
-
-def create_image_embedding_dataloader(
-    tar_url,
-    num_workers,
-    batch_size,
-    embeddings_url=None,
-    shard_width=None,
-    shuffle_num = None,
-    shuffle_shards = True,
-    resample_shards = False, 
-    handler=wds.handlers.warn_and_continue
-):
-    """
-    Convenience function to create an image embedding dataseta and dataloader in one line
-
-    :param tar_url: A url pointing to the tar files of the webdataset formatted as /path/to/webdataset/{0000..9999}.tar
-    :param num_workers: The number of workers to use for the dataloader
-    :param batch_size: The batch size to use for the dataloader
-    :param embeddings_url: Required if webdataset does not contain embeddings. A url pointing to the npy files of the embeddings. Should have the same number of shards as the webdataset.
-        Webdataset image keys should align with the index of the embedding. This means missing image indices must have a corresponding embedding of all zeros.
-    :param shard_width: The number of digits in the shard number. This is used to align the embedding index with the image index.
-        For example, if a file in the webdataset shard 3 is named 0003039.jpg, we know the shard width is 4 and the last three digits are the index.
-    :param shuffle_num: If not None, shuffle the dataset with this size buffer after sampling.
-    :param shuffle_shards: If true, shuffle the shards before sampling. This cannot be true if resample is true.
-    :param resample_shards: If true, resample webdataset shards with replacement. You need to set your own epoch size if this is true since it will resample infinitely.
-    :param handler: A webdataset handler.
-    """
-    ds = ImageEmbeddingDataset(
-        tar_url,
-        embeddings_url,
-        shard_width=shard_width,
-        shuffle_shards=shuffle_shards,
-        resample=resample_shards,
-        handler=handler
-    )
-    if shuffle_num is not None and shuffle_num > 0:
-        ds.shuffle(1000)
-    return wds.WebLoader(
-        ds,
-        num_workers=num_workers,
-        batch_size=batch_size,
-        prefetch_factor=2,  # This might be good to have high so the next npy file is prefetched
-        pin_memory=True,
-        shuffle=False
-    )

dalle2_pytorch/train.py

@@ -5,7 +5,7 @@ import torch
 from torch import nn
 from torch.cuda.amp import autocast, GradScaler
 
-from dalle2_pytorch.dalle2_pytorch import Decoder, DiffusionPrior
+from dalle2_pytorch.dalle2_pytorch import Decoder
 from dalle2_pytorch.optimizer import get_optimizer
 
 # helper functions
@@ -89,83 +89,7 @@ class EMA(nn.Module):
     def __call__(self, *args, **kwargs):
         return self.ema_model(*args, **kwargs)
 
-# diffusion prior trainer
-
-class DiffusionPriorTrainer(nn.Module):
-    def __init__(
-        self,
-        diffusion_prior,
-        use_ema = True,
-        lr = 3e-4,
-        wd = 1e-2,
-        max_grad_norm = None,
-        amp = False,
-        **kwargs
-    ):
-        super().__init__()
-        assert isinstance(diffusion_prior, DiffusionPrior)
-        ema_kwargs, kwargs = groupby_prefix_and_trim('ema_', kwargs)
-
-        self.diffusion_prior = diffusion_prior
-
-        # exponential moving average
-
-        self.use_ema = use_ema
-        if self.use_ema:
-            self.ema_diffusion_prior = EMA(diffusion_prior, **ema_kwargs)
-
-        # optimizer and mixed precision stuff
-
-        self.amp = amp
-
-        self.scaler = GradScaler(enabled = amp)
-
-        self.optimizer = get_optimizer(
-            diffusion_prior.parameters(),
-            lr = lr,
-            wd = wd,
-            **kwargs
-        )
-
-        # gradient clipping if needed
-
-        self.max_grad_norm = max_grad_norm
-
-    def update(self):
-        if exists(self.max_grad_norm):
-            self.scaler.unscale_(self.optimizer)
-            nn.utils.clip_grad_norm_(self.diffusion_prior.parameters(), self.max_grad_norm)
-
-        self.scaler.step(self.optimizer)
-        self.scaler.update()
-        self.optimizer.zero_grad()
-
-        if self.use_ema:
-            self.ema_diffusion_prior.update()
-
-    @torch.inference_mode()
-    def p_sample_loop(self, *args, **kwargs):
-        return self.ema_diffusion_prior.ema_model.p_sample_loop(*args, **kwargs)
-
-    @torch.inference_mode()
-    def sample(self, *args, **kwargs):
-        return self.ema_diffusion_prior.ema_model.sample(*args, **kwargs)
-
-    @torch.inference_mode()
-    def sample_batch_size(self, *args, **kwargs):
-        return self.ema_diffusion_prior.ema_model.sample_batch_size(*args, **kwargs)
-
-    def forward(
-        self,
-        *args,
-        divisor = 1,
-        **kwargs
-    ):
-        with autocast(enabled = self.amp):
-            loss = self.diffusion_prior(*args, **kwargs)
-        return self.scaler.scale(loss / divisor)
-
-# decoder trainer
+# trainers
 
 class DecoderTrainer(nn.Module):
     def __init__(
@@ -235,13 +159,12 @@ class DecoderTrainer(nn.Module):
         index = unet_number - 1
         unet = self.decoder.unets[index]
 
+        if exists(self.max_grad_norm):
+            nn.utils.clip_grad_norm_(unet.parameters(), self.max_grad_norm)
+
         optimizer = getattr(self, f'optim{index}')
         scaler = getattr(self, f'scaler{index}')
 
-        if exists(self.max_grad_norm):
-            scaler.unscale_(optimizer)
-            nn.utils.clip_grad_norm_(unet.parameters(), self.max_grad_norm)
-
         scaler.step(optimizer)
         scaler.update()
         optimizer.zero_grad()

dalle2_pytorch/train_vqgan_vae.py

@@ -1,277 +0,0 @@
-from math import sqrt
-import copy
-from random import choice
-from pathlib import Path
-from shutil import rmtree
-from PIL import Image
-
-import torch
-from torch import nn
-from torch.cuda.amp import autocast, GradScaler
-from torch.utils.data import Dataset, DataLoader, random_split
-
-import torchvision.transforms as T
-from torchvision.datasets import ImageFolder
-from torchvision.utils import make_grid, save_image
-
-from einops import rearrange
-
-from dalle2_pytorch.train import EMA
-from dalle2_pytorch.vqgan_vae import VQGanVAE
-from dalle2_pytorch.optimizer import get_optimizer
-
-# helpers
-
-def exists(val):
-    return val is not None
-
-def noop(*args, **kwargs):
-    pass
-
-def cycle(dl):
-    while True:
-        for data in dl:
-            yield data
-
-def cast_tuple(t):
-    return t if isinstance(t, (tuple, list)) else (t,)
-
-def yes_or_no(question):
-    answer = input(f'{question} (y/n) ')
-    return answer.lower() in ('yes', 'y')
-
-def accum_log(log, new_logs):
-    for key, new_value in new_logs.items():
-        old_value = log.get(key, 0.)
-        log[key] = old_value + new_value
-    return log
-
-# classes
-
-class ImageDataset(Dataset):
-    def __init__(
-        self,
-        folder,
-        image_size,
-        exts = ['jpg', 'jpeg', 'png']
-    ):
-        super().__init__()
-        self.folder = folder
-        self.image_size = image_size
-        self.paths = [p for ext in exts for p in Path(f'{folder}').glob(f'**/*.{ext}')]
-
-        print(f'{len(self.paths)} training samples found at {folder}')
-
-        self.transform = T.Compose([
-            T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
-            T.Resize(image_size),
-            T.RandomHorizontalFlip(),
-            T.CenterCrop(image_size),
-            T.ToTensor()
-        ])
-
-    def __len__(self):
-        return len(self.paths)
-
-    def __getitem__(self, index):
-        path = self.paths[index]
-        img = Image.open(path)
-        return self.transform(img)
-
-# main trainer class
-
-class VQGanVAETrainer(nn.Module):
-    def __init__(
-        self,
-        vae,
-        *,
-        num_train_steps,
-        lr,
-        batch_size,
-        folder,
-        grad_accum_every,
-        wd = 0.,
-        save_results_every = 100,
-        save_model_every = 1000,
-        results_folder = './results',
-        valid_frac = 0.05,
-        random_split_seed = 42,
-        ema_beta = 0.995,
-        ema_update_after_step = 2000,
-        ema_update_every = 10,
-        apply_grad_penalty_every = 4,
-        amp = False
-    ):
-        super().__init__()
-        assert isinstance(vae, VQGanVAE), 'vae must be instance of VQGanVAE'
-        image_size = vae.image_size
-
-        self.vae = vae
-        self.ema_vae = EMA(vae, update_after_step = ema_update_after_step, update_every = ema_update_every)
-
-        self.register_buffer('steps', torch.Tensor([0]))
-
-        self.num_train_steps = num_train_steps
-        self.batch_size = batch_size
-        self.grad_accum_every = grad_accum_every
-
-        all_parameters = set(vae.parameters())
-        discr_parameters = set(vae.discr.parameters())
-        vae_parameters = all_parameters - discr_parameters
-
-        self.optim = get_optimizer(vae_parameters, lr = lr, wd = wd)
-        self.discr_optim = get_optimizer(discr_parameters, lr = lr, wd = wd)
-
-        self.amp = amp
-        self.scaler = GradScaler(enabled = amp)
-        self.discr_scaler = GradScaler(enabled = amp)
-
-        # create dataset
-
-        self.ds = ImageDataset(folder, image_size = image_size)
-
-        # split for validation
-
-        if valid_frac > 0:
-            train_size = int((1 - valid_frac) * len(self.ds))
-            valid_size = len(self.ds) - train_size
-            self.ds, self.valid_ds = random_split(self.ds, [train_size, valid_size], generator = torch.Generator().manual_seed(random_split_seed))
-            print(f'training with dataset of {len(self.ds)} samples and validating with randomly splitted {len(self.valid_ds)} samples')
-        else:
-            self.valid_ds = self.ds
-            print(f'training with shared training and valid dataset of {len(self.ds)} samples')
-
-        # dataloader
-
-        self.dl = cycle(DataLoader(
-            self.ds,
-            batch_size = batch_size,
-            shuffle = True
-        ))
-
-        self.valid_dl = cycle(DataLoader(
-            self.valid_ds,
-            batch_size = batch_size,
-            shuffle = True
-        ))
-
-        self.save_model_every = save_model_every
-        self.save_results_every = save_results_every
-
-        self.apply_grad_penalty_every = apply_grad_penalty_every
-
-        self.results_folder = Path(results_folder)
-
-        if len([*self.results_folder.glob('**/*')]) > 0 and yes_or_no('do you want to clear previous experiment checkpoints and results?'):
-            rmtree(str(self.results_folder))
-
-        self.results_folder.mkdir(parents = True, exist_ok = True)
-
-    def train_step(self):
-        device = next(self.vae.parameters()).device
-        steps = int(self.steps.item())
-        apply_grad_penalty = not (steps % self.apply_grad_penalty_every)
-
-        self.vae.train()
-
-        # logs
-
-        logs = {}
-
-        # update vae (generator)
-
-        for _ in range(self.grad_accum_every):
-            img = next(self.dl)
-            img = img.to(device)
-
-            with autocast(enabled = self.amp):
-                loss = self.vae(
-                    img,
-                    return_loss = True,
-                    apply_grad_penalty = apply_grad_penalty
-                )
-
-
-                self.scaler.scale(loss / self.grad_accum_every).backward()
-
-            accum_log(logs, {'loss': loss.item() / self.grad_accum_every})
-
-        self.scaler.step(self.optim)
-        self.scaler.update()
-        self.optim.zero_grad()
-
-        # update discriminator
-
-        if exists(self.vae.discr):
-            discr_loss = 0
-            for _ in range(self.grad_accum_every):
-                img = next(self.dl)
-                img = img.to(device)
-
-                with autocast(enabled = self.amp):
-                    loss = self.vae(img, return_discr_loss = True)
-
-                    self.discr_scaler.scale(loss / self.grad_accum_every).backward()
-
-                accum_log(logs, {'discr_loss': loss.item() / self.grad_accum_every})
-
-            self.discr_scaler.step(self.discr_optim)
-            self.discr_scaler.update()
-            self.discr_optim.zero_grad()
-
-            # log
-
-            print(f"{steps}: vae loss: {logs['loss']} - discr loss: {logs['discr_loss']}")
-
-        # update exponential moving averaged generator
-
-        self.ema_vae.update()
-
-        # sample results every so often
-
-        if not (steps % self.save_results_every):
-            for model, filename in ((self.ema_vae.ema_model, f'{steps}.ema'), (self.vae, str(steps))):
-                model.eval()
-
-                imgs = next(self.dl)
-                imgs = imgs.to(device)
-
-                recons = model(imgs)
-                nrows = int(sqrt(self.batch_size))
-
-                imgs_and_recons = torch.stack((imgs, recons), dim = 0)
-                imgs_and_recons = rearrange(imgs_and_recons, 'r b ... -> (b r) ...')
-
-                imgs_and_recons = imgs_and_recons.detach().cpu().float().clamp(0., 1.)
-                grid = make_grid(imgs_and_recons, nrow = 2, normalize = True, value_range = (0, 1))
-
-                logs['reconstructions'] = grid
-
-                save_image(grid, str(self.results_folder / f'{filename}.png'))
-
-            print(f'{steps}: saving to {str(self.results_folder)}')
-
-        # save model every so often
-
-        if not (steps % self.save_model_every):
-            state_dict = self.vae.state_dict()
-            model_path = str(self.results_folder / f'vae.{steps}.pt')
-            torch.save(state_dict, model_path)
-
-            ema_state_dict = self.ema_vae.state_dict()
-            model_path = str(self.results_folder / f'vae.{steps}.ema.pt')
-            torch.save(ema_state_dict, model_path)
-
-            print(f'{steps}: saving model to {str(self.results_folder)}')
-
-        self.steps += 1
-        return logs
-
-    def train(self, log_fn = noop):
-        device = next(self.vae.parameters()).device
-
-        while self.steps < self.num_train_steps:
-            logs = self.train_step()
-            log_fn(logs)
-
-        print('training complete')

dalle2_pytorch/vqgan_vae.py

@@ -285,10 +285,6 @@ class ResnetEncDec(nn.Module):
     def get_encoded_fmap_size(self, image_size):
         return image_size // (2 ** self.layers)
 
-    @property
-    def last_dec_layer(self):
-        return self.decoders[-1].weight
-
     def encode(self, x):
         for enc in self.encoders:
             x = enc(x)
@@ -508,10 +504,6 @@ class ViTEncDec(nn.Module):
     def get_encoded_fmap_size(self, image_size):
         return image_size // self.patch_size
 
-    @property
-    def last_dec_layer(self):
-        return self.decoder[-3][-1].weight
-
     def encode(self, x):
         return self.encoder(x)
 
@@ -747,7 +739,7 @@ class VQGanVAE(nn.Module):
 
         # calculate adaptive weight
 
-        last_dec_layer = self.enc_dec.last_dec_layer
+        last_dec_layer = self.decoders[-1].weight
 
         norm_grad_wrt_gen_loss = grad_layer_wrt_loss(gen_loss, last_dec_layer).norm(p = 2)
         norm_grad_wrt_perceptual_loss = grad_layer_wrt_loss(perceptual_loss, last_dec_layer).norm(p = 2)

setup.py

@@ -10,7 +10,7 @@ setup(
       'dream = dalle2_pytorch.cli:dream'
     ],
   },
-  version = '0.2.2',
+  version = '0.0.82',
   license='MIT',
   description = 'DALL-E 2',
   author = 'Phil Wang',
@@ -24,22 +24,16 @@ setup(
   install_requires=[
     'click',
     'clip-anytorch',
-    'coca-pytorch>=0.0.5',
     'einops>=0.4',
     'einops-exts>=0.0.3',
-    'embedding-reader',
     'kornia>=0.5.4',
     'pillow',
-    'resize-right>=0.0.2',
-    'rotary-embedding-torch',
     'torch>=1.10',
     'torchvision',
     'tqdm',
     'vector-quantize-pytorch',
-    'x-clip>=0.4.4',
-    'youtokentome',
-    'webdataset>=0.2.5',
-    'fsspec>=2022.1.0'
+    'x-clip>=0.5.1',
+    'youtokentome'
   ],
   classifiers=[
     'Development Status :: 4 - Beta',

train_diffusion_prior.py

@@ -1,353 +0,0 @@
-import os
-import math
-import argparse
-import numpy as np
-
-import torch
-from torch import nn
-from embedding_reader import EmbeddingReader
-from dalle2_pytorch import DiffusionPrior, DiffusionPriorNetwork, load_diffusion_model, save_diffusion_model
-from dalle2_pytorch.optimizer import get_optimizer
-from torch.cuda.amp import autocast,GradScaler
-
-import time
-from tqdm import tqdm
-
-import wandb
-os.environ["WANDB_SILENT"] = "true"
-NUM_TEST_EMBEDDINGS = 100 # for cosine similarity reporting during training
-REPORT_METRICS_EVERY = 100 # for cosine similarity and other metric reporting during training
-
-
-def eval_model(model,device,image_reader,text_reader,start,end,batch_size,loss_type,phase="Validation"):
-    model.eval()
-    with torch.no_grad():
-        total_loss = 0.
-        total_samples = 0.
-
-        for emb_images, emb_text in zip(image_reader(batch_size=batch_size, start=start, end=end),
-                text_reader(batch_size=batch_size, start=start, end=end)):
-
-            emb_images_tensor = torch.tensor(emb_images[0]).to(device)
-            emb_text_tensor = torch.tensor(emb_text[0]).to(device)
-
-            batches = emb_images_tensor.shape[0]
-
-            loss = model(text_embed = emb_text_tensor, image_embed = emb_images_tensor)
-
-            total_loss += loss.item() * batches
-            total_samples += batches
-
-        avg_loss = (total_loss / total_samples)
-        wandb.log({f'{phase} {loss_type}': avg_loss})
-
-def report_cosine_sims(diffusion_prior,image_reader,text_reader,train_set_size,NUM_TEST_EMBEDDINGS,device):
-    diffusion_prior.eval()
-
-    cos = nn.CosineSimilarity(dim=1, eps=1e-6)
-
-    tstart = train_set_size
-    tend = train_set_size+NUM_TEST_EMBEDDINGS
-
-    for embt, embi in zip(text_reader(batch_size=NUM_TEST_EMBEDDINGS, start=tstart, end=tend), 
-            image_reader(batch_size=NUM_TEST_EMBEDDINGS, start=tstart, end=tend)):
-       # make a copy of the text embeddings for shuffling
-       text_embed = torch.tensor(embt[0]).to(device)
-       text_embed_shuffled = text_embed.clone()
-        # roll the text embeddings to simulate "unrelated" captions
-       rolled_idx = torch.roll(torch.arange(NUM_TEST_EMBEDDINGS), 1)
-       text_embed_shuffled = text_embed_shuffled[rolled_idx]
-       text_embed_shuffled = text_embed_shuffled / \
-           text_embed_shuffled.norm(dim=1, keepdim=True)
-       test_text_shuffled_cond = dict(text_embed=text_embed_shuffled)
-        # prepare the text embedding
-       text_embed = text_embed / text_embed.norm(dim=1, keepdim=True)
-       test_text_cond = dict(text_embed=text_embed)
-        # prepare image embeddings
-       test_image_embeddings = torch.tensor(embi[0]).to(device)
-       test_image_embeddings = test_image_embeddings / \
-           test_image_embeddings.norm(dim=1, keepdim=True)
-        # predict on the unshuffled text embeddings
-       predicted_image_embeddings = diffusion_prior.p_sample_loop(
-           (NUM_TEST_EMBEDDINGS, 768), text_cond=test_text_cond)
-       predicted_image_embeddings = predicted_image_embeddings / \
-           predicted_image_embeddings.norm(dim=1, keepdim=True)
-        # predict on the shuffled embeddings
-       predicted_unrelated_embeddings = diffusion_prior.p_sample_loop(
-           (NUM_TEST_EMBEDDINGS, 768), text_cond=test_text_shuffled_cond)
-       predicted_unrelated_embeddings = predicted_unrelated_embeddings / \
-           predicted_unrelated_embeddings.norm(dim=1, keepdim=True)
-        # calculate similarities
-       original_similarity = cos(
-           text_embed, test_image_embeddings).cpu().numpy()
-       predicted_similarity = cos(
-           text_embed, predicted_image_embeddings).cpu().numpy()
-       unrelated_similarity = cos(
-           text_embed, predicted_unrelated_embeddings).cpu().numpy()
-       predicted_img_similarity = cos(
-           test_image_embeddings, predicted_image_embeddings).cpu().numpy()
-       wandb.log({"CosineSimilarity(text_embed,image_embed)": np.mean(original_similarity),
-            "CosineSimilarity(text_embed,predicted_image_embed)":np.mean(predicted_similarity),
-            "CosineSimilarity(orig_image_embed,predicted_image_embed)":np.mean(predicted_img_similarity),
-            "CosineSimilarity(text_embed,predicted_unrelated_embed)": np.mean(unrelated_similarity),
-            "Cosine similarity difference":np.mean(predicted_similarity - original_similarity)})
-
-def train(image_embed_dim,
-          image_embed_url,
-          text_embed_url,
-          batch_size,
-          train_percent,
-          val_percent,
-          test_percent,
-          num_epochs,
-          dp_loss_type,
-          clip,
-          dp_condition_on_text_encodings,
-          dp_timesteps,
-          dp_normformer,
-          dp_cond_drop_prob,
-          dpn_depth,
-          dpn_dim_head,
-          dpn_heads,
-          save_interval,
-          save_path,
-          device,
-          RESUME,
-          DPRIOR_PATH,
-          config,
-          wandb_entity,
-          wandb_project,
-          learning_rate=0.001,
-          max_grad_norm=0.5,
-          weight_decay=0.01,
-          dropout=0.05,
-          amp=False):
-
-    # DiffusionPriorNetwork 
-    prior_network = DiffusionPriorNetwork( 
-            dim = image_embed_dim, 
-            depth = dpn_depth, 
-            dim_head = dpn_dim_head, 
-            heads = dpn_heads,
-            attn_dropout = dropout,
-            ff_dropout = dropout,
-            normformer = dp_normformer).to(device)
-    
-    # DiffusionPrior with text embeddings and image embeddings pre-computed
-    diffusion_prior = DiffusionPrior( 
-            net = prior_network, 
-            clip = clip, 
-            image_embed_dim = image_embed_dim, 
-            timesteps = dp_timesteps,
-            cond_drop_prob = dp_cond_drop_prob, 
-            loss_type = dp_loss_type, 
-            condition_on_text_encodings = dp_condition_on_text_encodings).to(device)
-
-    # Load pre-trained model from DPRIOR_PATH
-    if RESUME:
-        diffusion_prior=load_diffusion_model(DPRIOR_PATH,device)   
-        wandb.init( entity=wandb_entity, project=wandb_project, config=config) 
-
-    # Create save_path if it doesn't exist
-    if not os.path.exists(save_path):
-        os.makedirs(save_path)
-
-    # Get image and text embeddings from the servers
-    print("==============Downloading embeddings - image and text====================")
-    image_reader = EmbeddingReader(embeddings_folder=image_embed_url, file_format="npy")
-    text_reader  = EmbeddingReader(embeddings_folder=text_embed_url, file_format="npy")
-    num_data_points = text_reader.count
-
-    ### Training code ###
-    scaler = GradScaler(enabled=amp)
-    optimizer = get_optimizer(diffusion_prior.net.parameters(), wd=weight_decay, lr=learning_rate)
-    epochs = num_epochs
-
-    step = 0
-    t = time.time()
-
-    train_set_size = int(train_percent*num_data_points)
-    val_set_size = int(val_percent*num_data_points)
-    eval_start = train_set_size
-
-    for _ in range(epochs):
-
-        for emb_images,emb_text in zip(image_reader(batch_size=batch_size, start=0, end=train_set_size),
-                text_reader(batch_size=batch_size, start=0, end=train_set_size)):
-
-            diffusion_prior.train()
-            
-            emb_images_tensor = torch.tensor(emb_images[0]).to(device)
-            emb_text_tensor = torch.tensor(emb_text[0]).to(device)
-
-            with autocast(enabled=amp):
-                loss = diffusion_prior(text_embed = emb_text_tensor,image_embed = emb_images_tensor)
-                scaler.scale(loss).backward()
-
-            # Samples per second
-            step+=1
-            samples_per_sec = batch_size*step/(time.time()-t)
-            # Save checkpoint every save_interval minutes
-            if(int(time.time()-t) >= 60*save_interval):
-                t = time.time()
-
-                save_diffusion_model(
-                    save_path,
-                    diffusion_prior,
-                    optimizer,
-                    scaler,
-                    config,
-                    image_embed_dim)
-
-            # Log to wandb
-            wandb.log({"Training loss": loss.item(),
-                        "Steps": step,
-                        "Samples per second": samples_per_sec})
-            # Log cosineSim(text_embed,predicted_image_embed) - cosineSim(text_embed,image_embed)
-            # Use NUM_TEST_EMBEDDINGS samples from the test set each time
-            # Get embeddings from the most recently saved model
-            if(step % REPORT_METRICS_EVERY) == 0:
-                report_cosine_sims(diffusion_prior,
-                        image_reader,
-                        text_reader,
-                        train_set_size,
-                        NUM_TEST_EMBEDDINGS,
-                        device)
-                ### Evaluate model(validation run) ###
-                eval_model(diffusion_prior,
-                        device,
-                        image_reader,
-                        text_reader,
-                        eval_start,
-                        eval_start+NUM_TEST_EMBEDDINGS,
-                        NUM_TEST_EMBEDDINGS,
-                        dp_loss_type,
-                        phase="Validation")
-
-            scaler.unscale_(optimizer)
-            nn.utils.clip_grad_norm_(diffusion_prior.parameters(), max_grad_norm)
-
-            scaler.step(optimizer)
-            scaler.update()
-            optimizer.zero_grad()
-
-    ### Test run ###
-    test_set_size = int(test_percent*train_set_size) 
-    start=train_set_size+val_set_size
-    end=num_data_points
-    eval_model(diffusion_prior,device,image_reader,text_reader,start,end,batch_size,dp_loss_type,phase="Test")
-
-def main():
-    parser = argparse.ArgumentParser()
-    # Logging
-    parser.add_argument("--wandb-entity", type=str, default="laion")
-    parser.add_argument("--wandb-project", type=str, default="diffusion-prior")
-    parser.add_argument("--wandb-dataset", type=str, default="LAION-5B")
-    parser.add_argument("--wandb-arch", type=str, default="DiffusionPrior")
-    # URLs for embeddings 
-    parser.add_argument("--image-embed-url", type=str, default="https://mystic.the-eye.eu/public/AI/cah/laion5b/embeddings/laion2B-en/img_emb/")
-    parser.add_argument("--text-embed-url", type=str, default="https://mystic.the-eye.eu/public/AI/cah/laion5b/embeddings/laion2B-en/text_emb/")
-    # Hyperparameters
-    parser.add_argument("--learning-rate", type=float, default=1.1e-4)
-    parser.add_argument("--weight-decay", type=float, default=6.02e-2)
-    parser.add_argument("--dropout", type=float, default=5e-2)
-    parser.add_argument("--max-grad-norm", type=float, default=0.5)
-    parser.add_argument("--batch-size", type=int, default=10**4)
-    parser.add_argument("--num-epochs", type=int, default=5)
-    # Image embed dimension
-    parser.add_argument("--image-embed-dim", type=int, default=768)
-    # Train-test split
-    parser.add_argument("--train-percent", type=float, default=0.7)
-    parser.add_argument("--val-percent", type=float, default=0.2)
-    parser.add_argument("--test-percent", type=float, default=0.1)
-    # LAION training(pre-computed embeddings)
-    # DiffusionPriorNetwork(dpn) parameters
-    parser.add_argument("--dpn-depth", type=int, default=6)
-    parser.add_argument("--dpn-dim-head", type=int, default=64)
-    parser.add_argument("--dpn-heads", type=int, default=8)
-    # DiffusionPrior(dp) parameters
-    parser.add_argument("--dp-condition-on-text-encodings", type=bool, default=False)
-    parser.add_argument("--dp-timesteps", type=int, default=100)
-    parser.add_argument("--dp-normformer", type=bool, default=False)
-    parser.add_argument("--dp-cond-drop-prob", type=float, default=0.1)
-    parser.add_argument("--dp-loss-type", type=str, default="l2")
-    parser.add_argument("--clip", type=str, default=None)
-    parser.add_argument("--amp", type=bool, default=False)
-    # Model checkpointing interval(minutes)
-    parser.add_argument("--save-interval", type=int, default=30)
-    parser.add_argument("--save-path", type=str, default="./diffusion_prior_checkpoints")
-    # Saved model path 
-    parser.add_argument("--pretrained-model-path", type=str, default=None)
-
-    args = parser.parse_args()
-
-    config = ({"learning_rate": args.learning_rate,
-        "architecture": args.wandb_arch,
-        "dataset": args.wandb_dataset,
-        "weight_decay":args.weight_decay,
-        "max_gradient_clipping_norm":args.max_grad_norm,
-        "batch_size":args.batch_size,
-        "epochs": args.num_epochs,
-        "diffusion_prior_network":{"depth":args.dpn_depth,
-        "dim_head":args.dpn_dim_head,
-        "heads":args.dpn_heads,
-        "normformer":args.dp_normformer},
-        "diffusion_prior":{"condition_on_text_encodings": args.dp_condition_on_text_encodings,
-        "timesteps": args.dp_timesteps,
-        "cond_drop_prob":args.dp_cond_drop_prob,
-        "loss_type":args.dp_loss_type,
-        "clip":args.clip}
-        })
-
-    RESUME = False
-    # Check if DPRIOR_PATH exists(saved model path)
-    DPRIOR_PATH = args.pretrained_model_path
-    if(DPRIOR_PATH is not None):
-        RESUME = True
-    else:
-        wandb.init(
-          entity=args.wandb_entity,
-          project=args.wandb_project,
-          config=config)
-
-    # Obtain the utilized device.
-
-    has_cuda = torch.cuda.is_available()
-    if has_cuda:
-        device = torch.device("cuda:0")
-        torch.cuda.set_device(device)
-
-    # Training loop
-    train(args.image_embed_dim,
-          args.image_embed_url,
-          args.text_embed_url,
-          args.batch_size,
-          args.train_percent,
-          args.val_percent,
-          args.test_percent,
-          args.num_epochs,
-          args.dp_loss_type,
-          args.clip,
-          args.dp_condition_on_text_encodings,
-          args.dp_timesteps,
-          args.dp_normformer,
-          args.dp_cond_drop_prob,
-          args.dpn_depth,
-          args.dpn_dim_head,
-          args.dpn_heads,
-          args.save_interval,
-          args.save_path,
-          device,
-          RESUME,
-          DPRIOR_PATH,
-          config,
-          atgs.wandb_entity,
-          args.wandb_project,
-          args.learning_rate,
-          args.max_grad_norm,
-          args.weight_decay,
-          args.dropout,
-          args.amp)
-
-if __name__ == "__main__":
-  main()