Sporksmith's Blog

If you haven’t already, you might want to read Part 1: Labeling.

The next feature I needed was, given a desired position for Mr. Monkey, to come up with a photo with Mr. Monkey in that position. There’s a number of ways to come at this problem, but let’s first consider the problem of taking any of the labeled photos, and making it match the desired position and orientation. We can actually come pretty close for any of the photos, using the following transformations:

• Scale. Scale the photo so that Mr. Monkey in the photo is exactly the desired size.
• Pan. Move the photo so that Mr. Monkey in the photo is at exactly the desired location.
• Rotate. Unfortunately, I don’t have access to the magic CSI computers to rotate Mr. Monkey in the photo freely in all dimensions, so it’s not generally possible to make Mr. Monkey in the photo face exactly the desired direction. I can’t make him face the camera if he’s facing away from the camera in the photo, but if he’s facing the camera straight-on, I can rotate the photo to make him lean to the left. There’s some annoying 3d geometry required to do this in a fully general way, of which we shall not speak.
• Mirror. Here’s one more dirty trick I can pull to make Mr. Monkey face closer to the desired location. If he’s facing to the left, I can make him face to the right instead by mirroring the photo.

While we can force any of the photos to sort-of match the desired position and rotation, it’s not going to look equally good for any given photo. Here are some of the problems we can end up with:

• Rotation still doesn’t match closely. There’s only so much I can do to change which direction Mr. Monkey is facing.
• Some of the picture is no longer in frame. After applying the above transformations (scaling, panning, and rotating), some of the photo may be effectively cropped off (out of frame). If Mr. Monkey is talking to someone in the photo, cropping that person out of the photo causes it to no longer make sense.
• Some of the frame might be unused. If Mr. Monkey was all the way on the right side of the photo, and we moved him to be all the way on the left side of the frame, not only is the picture out of frame, but a lot of the frame is unused, revealing a boring blank background. This can also happen if we scale the photo down smaller than the frame size.
• Pixellation from scaling. Mr. Monkey starts looking chunky if we scale him from fifty pixels to two-hundred…twunkey.

So, to find the best match, we just see which transformations are needed to make each candidate photo match, see which of those had the least of those undesired effects, and then go with that one. There are smarter things we could do here, but this is good enough for now. I’ve thought about using a more sophisticated data structure so that I don’t need to consider the entire photo pool for each match, but for now the simple way is good enough: it takes a little under one second to find the best match and load the selected photo from disk and apply the transformations.

A couple notes about the video: I had the mirroring transformation turned off, so none of the matches demonstrated are mirrored. Of course, the mirroring is typically only noticeable if there happens to be some text in the photo. Second, if you are somewhat obsessed with arithmetic, you might have noticed that the components of the match score don’t add up to the total score. This is because there is a weighting applied to each sub-score before adding them all together. For example, in the current settings, the fraction out-of-frame is multiplied by 0.1 before adding it to the other scores. In the actual animation, having a large portion of an individual photo go out of frame doesn’t matter much, because it won’t be on-screen long enough for you to notice anyways; it’s more important that the rotations match, and that it isn’t scaled too horrifically.

Next time: Animation!