This past week, I’ve been playing around with more image processing and generation techniques. In particular, I implemented the neural style transfer algorithm by Gatys, Ecker, and Bethge in PyTorch following this tutorial. The paper and technique have been around for a few years, but it wasn’t until now that I have access to a GPU here at Recurse. This was so much fun to implement and experiment with!
My GitHub repo contains instructions on setup and usage, as well as a directory containing many results, if you would like to try it out and explore for yourself!
Model Overview
Neural style transfer takes two images as input and applies the style of one image onto the content of the other. In the example below, the first image is the style input, the second image is the content input, and the third image is the result of the style transfer. (The style image used here is one of my favorite paintings: Nocturne in Black and Gold, the Falling Rocket by James Abbott McNeill Whistler.)
The approach builds off of the VGG-19, a convolutional neural network pretrained on millions of images. It’s 19 layers deep and built by the Visual Geometry Group, hence VGG-19.
For neural style transfer, we modify the network architecture as such: we insert a content loss layer, using mean squared error, after the fourth convolutional layer; and insert style loss layers, using mean squared error on normalized gram matrices, after the first five convolutional layers.
Results
These are some of my favorite images that resulted from my explorations. Note that this is a curated collection of results.
