» About     » Archive     » Submit     » Authors     » Search     » Random     » Specials     » Statistics     » Forum     » RSS Feed     Updates Daily

No. 3567: Optical Flowfield

First | Previous | 2019-02-23 | Next | Latest

Optical Flowfield

First | Previous | 2019-02-23 | Next | Latest

Permanent URL: https://mezzacotta.net/garfield/?comic=3567

Strip by: David Morgan-Mar

{Six rows, each row comprising, on the left, an original Garfield strip; in the middle, a deformed version of that strip; on the right, a collection of colourful blobs}

The author writes:

There have been several "Spot the Difference" strips in SRoMG, as submitters point out how some strips are almost identical to previously published strips.

But just how identical are they?

Image processing to the rescue!

For this experiment, I chose three of the published "Spot the Difference" strips:

The first one I chose because the strips are, apart from colouring, very similar. The second I chose because there was a bit more visual difference, especially in the first two panels, in which Garfield appears in one strip but not the other, and Jon's and Irma's poses are different in panel 2. The third I chose because although the joke is virtually the same, the two strips are structurally a lot more different, with Odie in one and not the other, and Jon on the other side of Garfield in panel 2.

The original six strips are shown in the left column of the above image, sorted into their matching pairs, one above the other.

To analyse these pairs of strips, I used an optical flow algorithm to warp each strip of each pair to produce the best visual match to the other strip in the pair. Optical flow attempts to identify features in a sequence of images that represent the same object, and then determine the motion vectors on the pixel grid that map any given feature in one image to its corresponding feature (i.e. representing the same object) in the other image. Optical flow is typically used in video processing applications, to identify and track moving objects in scenes, or the motion of the background. Although usually used on video frames, optical flow can just as easily be applied to a pair of photos - if the photos are similar enough.

So how did we do? The second column of strips in the image above shows the results of warping the other strip in the pair to match the strip in the first column. The third column of the above image shows a visualisation of the warp vectors at each pixel in the warped comic strip in each row. To interpret this visualisation, red means that the pixel in the unwarped second strip has been moved to the right to produce the warped strip, yellow means the pixel has been moved down, green means it has been moved to the left, and blue means it has been moved up. Intermediate shades represent intermediate angles: orange means the pixel has been moved right and down, for example. Brighter colours mean the pixel has moved further, and darker colours mean the pixel has moved less; black indicates the pixel has not moved at all.

Now, let's look at the results in detail. To make things easier, I'll refer to each strip consistently as follows: Zip1 (purple background), Zip2 (blue background), Irma1 (orangey background), Irma2 (pink background), Haha1 (green background), Haha2 (pink background).

The first pair of strips (Zip1, Zip2) illustrate the optical flow principle in a relatively simple manner, since the original strips were so similar to begin with. In the first panel, notice how Jon's hand is higher in Zip1 and lower in Zip2. The optical flow algorithm has identified Jon's hand as the same object and moved it to match. In the warped Zip2, Jon's hand has been moved up to match its position in Zip1, and we can see the blue area in the warp map indicating the upward movement. Reciprocally, in the warped Zip1, Jon's hand has been moved down to match its position in Zip2, and we can see the yellow area in the warp map indicating the upward movement. The up/down motion extends to the bottom edge of the speech balloon in both cases. The left edge of Jon's speech balloon has moved left in warped Zip2 (the green area in the warp map), and right in warped Zip1 (the red area in the warp map).

In the second panel, we can see that the "Zip Zip" sound effect in Zip1 uses a thinner font than in Zip2, and the "Z" has sharp corners in Zip1, but squared corners in Zip2. The optical flow has warped the Zip1 font to become fatter, and bent the diagonal lines of the "Z" to partly approximate the squared corners on Zip2, while the Zip2 font has become thinner and the squared corners have been stretched out to make them pointier. The resulting warp vectors in the map are all over the place, as things have been pulled in different directions in relatively close proximity.

The major feature in panel 3 is that Jon is further left in Zip1 and right in Zip2. In the warped Zip2, Jon has been pulled left (the large green area in the warp map), while in the warped Zip1, he has been pushed right (the red area in the warp map). The algorithm has done quite a good job on Jon, as his position in each warped image is a good match for his position in the unwarped partner image. Garfield's pose is almost the same so he doesn't move much. And there's an interesting thing where the algorithm tries to match the puffs of cloud behind Garfield.

In the Irma strips, Irma is further right in panels 1 and 2 of Irma1 than Irma2, and she ends up moving significantly to the right in warped Irma2 (the two large red blobs in the warp map), and left in warped Irma1 (the large green blobs). There's a very interesting thing happening with Jon and Garfield in the first panel. The algorithm has identified Jon in Irma1 with Garfield in Irma 2, and Jon in Irma 2 with his coffee cup in Irma1. You can see this best in the warped Irma1, in which, in the first two panels, Jon has been warped to the position of Garfield and, because Jon is taller, his head has been shrunk considerably. Garfield is represented entirely by Jon's stomach, which is probably appropriate. Also in these panels, Jon's maroon coffee mug has been stretched to fill the place where Jon's shirt is in Irma2. Jon's head is missing from panel 1, but in panel 2, Irma's hand has been stretched out horizontally to the right to serve as a proxy for Jon's head! (The big red area in the middle of the warp map.) Conversely, in the warped Irma2, Garfield has been shifted left to occupy the place where Jon stands in Irma1, and Jon has been shrunk down almost to the size of his maroon coffee mug. And Jon's head in warped Irma2 is now in the position of Irma's hand in Irma1.

Panel 3 is easier to interpret, as the poses are more similar in both strips. The main feature to look for is Jon's hair in Irma2 being stretched upwards to partly fill the gap between Irma's and Garfield's speech balloons in Irma1.

In the Haha strips, we see Odie and Jon being stretched to fill each other's positions in panel 1. Garfield and Jon adopt each other's poses in warped Haha2 panel 2, with Jon's licence turning into Odie. While in warped Haha1 panel 2, Jon becomes Garfield, and Odie and Garfield together attempt to warp into Jon. And things go bizarre in panel 3, where Garfield's stomach in warped Haha2 attempts to stretch out to fill his own upper body, while his head tries to vanish.

There are plenty more interesting features to note in all of these strips and warp maps, but I'll leave further interpretation as an exercise for the reader.

I should point out that optical flow struggles when there are actually no sensible correspondences between the images it is given. So given the inputs, it's done a reasonable job in all these cases. In terms of analysing how similar these "Spot the Difference" strips are: the Zip strips really are quite similar, but in optical flow terms, the other two pairs come out as pretty different.

Original strips: 2001-12-03, 2002-12-07, 2006-08-31, 2009-11-09, 2011-07-25, 2014-02-26.