CycleGANs to Create Computer-Generated Art

This loss actually consists of a few different components.

First, we calculate the GAN loss for the abstract painting generator (in the “cycle” we are in the yellow shaded portion of the figure below).

This is done by taking a real forest image and putting it through the abstract painting generator to generate a fake abstract painting.

The fake abstract painting is then put through the the abstract painting discriminator.

We can think of the output of the discriminator as being the probability that the picture is a real abstract painting.

The output of the discriminator is then evaluated with binary cross-entropy loss.

This is the abstract paining GAN loss.

GAN loss for abstract painting generatorSecond, we calculate the GAN loss for the forest image generator (in the cycle we are in the yellow shaded portion of the figure below).

This is done by taking a real abstract painting and putting it through the forest image generator to generate a fake forest image.

The fake forest image is then put through the the forest image discriminator.

We can think of the output of the discriminator as being the probability that the picture is a real forest image.

The output of the discriminator is then evaluated with binary cross-entropy loss.

This is the forest image GAN loss.

GAN loss for forest image generatorOkay great now we have a fake abstract painting and a fake forest image.

As explained above in the high-level overview, we now have to take those fake images and reconstruct them back to the corresponding real images.

So, third, we take the fake abstract painting and put it through the forest image generator to generate the reconstructed-original real forest image (in the cycle we are in the yellow shaded portion of the figure below).

We evaluate this reconstructed forest image against the real forest image with L1 loss (Note, here, the L1 loss is typically multiplied by some constant so it is on the same scale as the other calculated losses).

This is the forest image cycle-consistency loss.

Lastly, we take the fake forest image and put it through the abstract painting generator to generate the reconstructed abstract painting (in the cycle we are in the yellow shaded portion of the figure below).

We evaluate this reconstructed abstract painting against the real abstract painting with L1 loss (Note, here, the L1 loss is typically multiplied by some constant so it is on the same scale as the other calculated losses).

This is the abstract painting cycle-consistency loss.

Now we are almost ready to calculated the generator loss (yes, all of this work was for one number).

There is one more loss we need to calculate, which is the identity mapping.

The identity mapping loss is calculated because, as can be seen in the figure below, this step has been shown to preserve the colors of the original image.

First, we put the real forest image through the forest image generator and calculate the L1 loss for the real forest image and the generated forest image.

This is the forest image identity mapping loss (Note, this loss is also multiplied by constants).

Second, we do the same thing for the abstract paintings, where we put the real abstract paining through the abstract painting generator.

This is the abstract painting identity mapping loss.

Figure 9 from the CycleGAN paper.

We are finally ready to calculate the generator loss, which are all of the loss components added up:Generator Loss = abstract paining GAN loss + forest image GAN loss + forest image cycle-consistency loss + abstract painting cycle-consistency loss + forest image identity mapping loss + abstract painting identity mapping lossThe weights of the generators are then updated with respect to this loss!2.

Train the discriminatorsNow that we have updated the weights of the generators, next we need to train the discriminators.

First, we update the weights of the abstract painting discriminator.

We put a real abstract painting through the abstract painting discriminator and take this output and evaluate it with binary cross-entropy loss.

Then, we take the previously generated fake abstract painting, put it through the abstract painting discriminator, and also evaluate it with binary cross-entropy loss.

We then take the average of these two losses.

This is the abstract painting discriminator loss.

The weights of the abstract painting discriminator are updated with respect to this loss.

Second, we update the weights of the forest image discriminator.

We put a real forest image through the forest image discriminator and take this output and evaluate it with binary cross-entropy loss.

Then, we take the previous generated fake forest image, put it through the forest image discriminator, and also evaluate it with binary cross-entropy loss.

We then take the average of these two losses.

This is the forest image discriminator loss.

The weights of the forest image discriminator are updated with respect to this loss.

The ArchitectureSo far I have mentioned that these generators and discriminators exist, but I have yet to say the exact architecture of these neural networks.

Well there are, of course, a number of options for the architecture of these networks, but here I will briefly mention the architectures used to generate abstract paintings of forests.

The details of these architectures may be viewed in the code.

The generators come from an architecture previously used for style-transfer and super-resolution (see Johnson et al.

).

The network architecture basically consists of a bunch of ResNet blocks that downsample the grid size, keep the grid size constant, and then upsample the grid size.

See the figure below for a diagram of the architecture.

Generator architecture from Johnson et al.

For the discriminators, we used a PatchGAN, which basically tries to classify if each N x N patch (here, 70 x 70) of an image is real or fake.

See the code and this paper for more details of the PatchGAN.

Computer-Generated Forest ImagesAfter training for awhile, lets see how our computer-generated abstract forest images look!.Below are a few examples, but you can view all of them on my GitHub.

Not too bad!ConclusionsAs you can see above, it does indeed appear that computers can generate art thanks to CycleGANs in this case.

Again, the really cool part of CycleGANs is that you don’t need paired images in the dataset.

It is quite amazing that this is possible!.I will be curious to see if unpaired machine learning is possible in other fields, such as natural language processing (maybe this already exists!).

.. More details

Leave a Reply