In Episode 6, Scientist mentioned the concept of Neural Style Transfer to Poppy and her classmates. So what is Neural Style Transfer?
Neural Artistic Style Transfer is one of the coolest techniques to come out of the machine learning craze. Here’s an example of what it looks like:
There are three aspects to Neural Style Transfer, or NST. There is the original photo, called the “content image.” The reference photo is the “style image.” Finally, the content of the “content image” is stylized with the style of the “style image,” creating the resulting “pastiche.” A pastiche literally means an image in the style of another work.
So, how does this process even work? A neural network is used. More specifically, a convolutional neural network that has already been trained to recognize and classify images.
First, the pastiche is initialized as simple noise. Then, the content and style images are passed through the neural network layers. Each layer and iteration through the neural network returns values, which are then used to find the “style loss” and “content loss.”
The “style loss” shows how close the style of the pastiche is to the style of the style image. As you can see below, each picture depicts style loss, increasing from left to right. The end seems to be mostly garbage, so the actual style used would be of a layer beforehand.
The “content loss” shows how close the content of the pastiche is to the content of the content image. The content loss is show below, increasing from left to right. The image gets more distorted through each iteration, so the resulting image used would be of one before the last one.
Each iteration through the neural network changes certain values within the process of style transfer to lead to making these two losses as small as possible. With each iteration, the pastiche changes a little bit, getting closer and closer to the final stylized image. When these two loss values are as small as possible, then we get the final stylized image.