Phil Wang
|
a6cdbe0b9c
|
relax learning rate constraint, as @rom1504 wants to try a higher one
|
2022-07-06 18:09:11 -07:00 |
|
Phil Wang
|
e928ae5c34
|
default the device to the device that the diffusion prior parameters are on, if the trainer was never given the accelerator nor device
|
2022-07-06 12:47:48 -07:00 |
|
Phil Wang
|
1bd8a7835a
|
attempting to fix issue with deepspeed fp16 seeing overflowing gradient
|
2022-07-06 08:27:34 -07:00 |
|
Phil Wang
|
f33453df9f
|
debugging with Aidan
|
2022-07-05 18:22:43 -07:00 |
|
Phil Wang
|
1e4bb2bafb
|
cast long as float before deriving sinusoidal pos emb
|
2022-07-05 18:01:22 -07:00 |
|
Phil Wang
|
ee75515c7d
|
remove forcing of softmax in f32, in case it is interfering with deepspeed
|
2022-07-05 16:53:58 -07:00 |
|
Phil Wang
|
ec68243479
|
set ability to do warmup steps for each unet during training
|
2022-07-05 16:24:16 -07:00 |
|
Phil Wang
|
3afdcdfe86
|
need to keep track of training steps separately for each unet in decoder trainer
|
2022-07-05 15:17:59 -07:00 |
|
Phil Wang
|
b9a908ff75
|
bring in two tricks from the cogview paper for reducing the chances of overflow, for attention and layernorm
|
2022-07-05 14:27:04 -07:00 |
|
Phil Wang
|
e1fe3089df
|
do bias-less layernorm manually
|
2022-07-05 13:09:58 -07:00 |
|
Phil Wang
|
ec5a77fc55
|
0.15.4
|
2022-07-02 08:56:34 -07:00 |
|
Phil Wang
|
3d23ba4aa5
|
add ability to specify full self attention on specific stages in the unet
|
2022-07-01 10:22:07 -07:00 |
|
Phil Wang
|
282c35930f
|
0.15.2
|
2022-07-01 09:40:11 -07:00 |
|
Phil Wang
|
7b0edf9e42
|
allow for returning low resolution conditioning image on forward through decoder with return_lowres_cond_image flag
|
2022-07-01 09:35:39 -07:00 |
|
Phil Wang
|
a922a539de
|
bring back convtranspose2d upsampling, allow for nearest upsample with hyperparam, change kernel size of last conv to 1, make configurable, cleanup
|
2022-07-01 09:21:47 -07:00 |
|
Phil Wang
|
8f2466f1cd
|
blur sigma for upsampling training was 0.6 in the paper, make that the default value
|
2022-06-30 17:03:16 -07:00 |
|
Phil Wang
|
908ab83799
|
add skip connections for all intermediate resnet blocks, also add an extra resnet block for memory efficient version of unet, time condition for both initial resnet block and last one before output
|
2022-06-29 08:16:58 -07:00 |
|
Phil Wang
|
46a2558d53
|
bug in pydantic decoder config class
|
2022-06-29 07:17:35 -07:00 |
|
Phil Wang
|
6a11b9678b
|
bring in the skip connection scaling factor, used by imagen in their unets, cite original paper using it
|
2022-06-26 21:59:55 -07:00 |
|
Phil Wang
|
b90364695d
|
fix remaining issues with deriving cond_on_text_encodings from child unet settings
|
2022-06-26 21:07:42 -07:00 |
|
Phil Wang
|
032e83b0e0
|
nevermind, do not enforce text encodings on first unet
|
2022-06-26 12:45:05 -07:00 |
|
Phil Wang
|
2e85e736f3
|
remove unnecessary decoder setting, and if not unconditional, always make sure the first unet is condition-able on text
|
2022-06-26 12:32:17 -07:00 |
|
Phil Wang
|
4b994601ae
|
just make sure decoder learning rate is reasonable and help out budding researchers
|
2022-06-23 11:29:28 -07:00 |
|
Phil Wang
|
c8422ffd5d
|
fix EMA updating buffers with non-float tensors
|
2022-06-22 07:16:39 -07:00 |
|
Phil Wang
|
0021535c26
|
move ema to external repo
|
2022-06-20 11:48:32 -07:00 |
|
Phil Wang
|
f545ce18f4
|
be able to turn off p2 loss reweighting for upsamplers
|
2022-06-20 09:43:31 -07:00 |
|
Phil Wang
|
fc7abf624d
|
in paper, blur sigma was 0.6
|
2022-06-20 09:05:08 -07:00 |
|
Phil Wang
|
138079ca83
|
allow for setting beta schedules of unets differently in the decoder, as what was used in the paper was cosine, cosine, linear
|
2022-06-20 08:56:37 -07:00 |
|
Phil Wang
|
461b91c5c1
|
also merge distributed training code for decoder, thanks to @Veldrovive
|
2022-06-19 09:26:44 -07:00 |
|
Phil Wang
|
e37072a48c
|
0.10.0
|
2022-06-19 08:50:53 -07:00 |
|
Phil Wang
|
6651eafa93
|
one more residual, after seeing good results on unconditional generation locally
|
2022-06-16 11:18:02 -07:00 |
|
Phil Wang
|
e6bb75e5ab
|
fix missing residual for highest resolution of the unet
|
2022-06-15 20:09:43 -07:00 |
|
Phil Wang
|
b7f9607258
|
make memory efficient unet design from imagen toggle-able
|
2022-06-15 13:40:26 -07:00 |
|
Phil Wang
|
2219348a6e
|
adopt similar unet architecture as imagen
|
2022-06-15 12:18:21 -07:00 |
|
Phil Wang
|
9eea9b9862
|
add p2 loss reweighting for decoder training as an option
|
2022-06-14 10:58:57 -07:00 |
|
Phil Wang
|
5d958713c0
|
fix classifier free guidance for image hiddens summed to time hiddens, thanks to @xvjiarui for finding this bug
|
2022-06-13 21:01:50 -07:00 |
|
Phil Wang
|
350a3d6045
|
0.6.16
|
2022-06-06 08:45:46 -07:00 |
|
Phil Wang
|
934c9728dc
|
some cleanup
|
2022-06-04 16:54:15 -07:00 |
|
Phil Wang
|
ce4b0107c1
|
0.6.13
|
2022-06-04 13:26:57 -07:00 |
|
Phil Wang
|
22cc613278
|
ema fix from @nousr
|
2022-06-03 19:44:36 -07:00 |
|
Phil Wang
|
708809ed6c
|
lower beta2 for adam down to 0.99, based on https://openreview.net/forum?id=2LdBqxc1Yv
|
2022-06-03 10:26:28 -07:00 |
|
Phil Wang
|
9cc475f6e7
|
fix update_every within EMA
|
2022-06-03 10:21:05 -07:00 |
|
Phil Wang
|
ffd342e9d0
|
allow for an option to constrain the variance interpolation fraction coming out from the unet for learned variance, if it is turned on
|
2022-06-03 09:34:57 -07:00 |
|
Phil Wang
|
8cc278447e
|
just cast to right types for blur sigma and kernel size augs
|
2022-06-02 11:21:58 -07:00 |
|
Phil Wang
|
38cd62010c
|
allow for random blur sigma and kernel size augmentations on low res conditioning (need to reread paper to see if the augmentation value needs to be fed into the unet for conditioning as well)
|
2022-06-02 11:11:25 -07:00 |
|
Phil Wang
|
a851168633
|
make youtokentome optional package, due to reported installation difficulties
|
2022-06-01 09:25:35 -07:00 |
|
Phil Wang
|
1ffeecd0ca
|
lower default ema beta value
|
2022-05-31 11:55:21 -07:00 |
|
Phil Wang
|
3df899f7a4
|
patch
|
2022-05-31 09:03:43 -07:00 |
|
Phil Wang
|
6f8b90d4d7
|
add packaging package
|
2022-05-30 11:45:00 -07:00 |
|
Phil Wang
|
b588286288
|
fix version
|
2022-05-30 11:06:34 -07:00 |
|