# Generative Models ## Style Transfer Three approaches to Style transfer 1. Supervised Learning - Impractical - need pairs of images of original and stylized images (yeah lots of pairs) 2. Neural Style Transfer - more popular - Model extract style from image 1 and content from image 2 and create an image 3 with style of image 1 and content of image 2 - we are only using 2 images. model is "trained" until satisfactory results are obtained 3. Fast Neural Style Transfer - newer and better 1. "Perceptual losses for real-time style transfer and super solution" ### Neural Style Transfer ![[Pasted image 20210321013002.png]] 1. use a pretrained CNN (such as VGG-19) to extract content from content image 1. You can use a few layers before the last layer to extract content 2. use another pretrained CNN to extract style from the style image. However, there is style information at each layer 1. You can use the early layers of each convolutional block to extract the style 3. Initialize the generated image with the content image 4. compare the content of the generated image and the content of the content image to calculate content loss 5. compare the style of the generated image and the style of the style image to calculate style loss 6. calculate a total loss 7. update the generated image based on gradient descent on total loss ### Loss Function #### Content loss Content loss is a straight up sum of square of difference between the neural network output of generated image and content image #### Style loss Style loss is a sum of sum square of difference between the gram matrices of the style ouputs of the generated image and the style image. Few points - You need to take a sum of the losses of each style matrix - you need to do a reduce sum of square of differences between two gram matrices - gram matrices are a special matrix algebra technique to reduce a multidimensional tensor - calculating gram matrix and reduce sum can all be done by "Einstein sum notation" feature of tensorflow (similar to numpy) ### Total Loss - Total loss is a weighted sum of content and style losses ### Total Variational Loss The model trained with the total loss listed above looks like garbage. This is because of additional noise introduced in the process. You can reduce the noise by adding an regularization term on high frequency components