Intro to Computational Media - Pixels
Circles and Noses
Link to pixels homework [PoseNet Snake] in the p5 web editor.
First, an aside: the thing I’m happiest with having made in p5 in recent weeks is this New York City locale invitation for Applications. I treated it as an opportunity to try wrapping my head around these notes on arrays and objects better, and I think it helped.
How I arrived at the idea for this week’s homework: After the brief demonstration of PoseNet in class, I knew I wanted to try using it in an assignment. Combine that with the fact that I’ve been watching more Coding Train videos off the track of the ICM syllabus (for two reasons, first that I want to dabble in more topics, second that my other YouTube recs are just terrible of late), and that’s how the game Snake comes into the picture.
So, I wanted to do something with video and PoseNet while wrapping in Snake. More specifically, I wanted to make a version of Snake where the Snake’s position is controlled by the position of the player’s nose. I *also* wanted to see if my video tinkering could break free from the regular columns and rows grid.
The grid ideas I explored didn’t amount to much. I came across a fabulous write up by Ahmad Moussa on drawing a hexagonal grid in p5, and I’d like to revisit it (possibly with the sequencer I want to make for our sound assignment). My attempt to combine this hex grid technique with my knowledge of manipulating video/the pixel array did not go well–I think the long and the short of it is that I needed to think more about how I’m grabbing the color for each pixel corresponding to a given hexagon.
This melty, alpha-additive webcam mirror made of circles I made as another exploration actually sort of works, but the performance is terrible. Plus, I was far enough down the track on the PoseNet and Snake stuff by the time I made this to know there was likely no way I’d be able to figure out combining any kind of offset grid with my other aims.
If I’m proud of any one small trick I did with this week’s homework, it was using the round() function to keep the red rectangle drawn over the user’s nose on-grid. That’s a small tweak relative to the PoseNet examples I followed, but it’s exactly the sort of mathy aesthetics thing I definitely wouldn’t have been able to think through a few weeks ago.
As for modifying the pixels, since I wanted the game of Snake to be playable, I didn’t want there to be too much happening in the background of the game so as to be distracting. The idea I settled on is also an attempt to give the user some extra feedback that was born of a bug: when the Snake eats a piece of food, that food’s randomly assigned color becomes a filter over the pixelated webcam feed (one large rect(), really) that persists until the next piece of food is eaten, and so on.
That aforementioned bug came about when I was trying to keep the Snake’s head tied to the user’s nose: at one point, I had a Snake object following the PoseNet position of the nose perfectly, but it wouldn’t grow. You could have the Snake eat food by moving your nose to the food’s position, but the Snake would remain length one. The color change, at least, would be a way of signaling that the “eat” action was successful, but I like how it looks and kept it. In the Monday ICM help session Sam Heckle helped me figure out that the growth problem stemmed from new Snakes being created on each loop.
Okay, so how is this game different from the Coding Train Snake redux besides the webcam feed? Well, it’s worse!
I’m joking, but really–the PoseNet estimation is pretty jittery, which would make for erratic Snake movement and probably a lot of inexplicable game overs or resets if I got it working and kept self-collision enabled. The “game” I inadvertently made with just moving your nose around to eat food is sort of cute, but it could also be made in a much simpler fashion than piggybacking off of Shiffman’s Snake code (you’d just check to see if pose.nose.x and pose.nose.y equal the coordinates of the food on each frame and voila). With that all in mind I arrived at the game’s new control scheme: treating the user’s nose position like arrow keys.
Limiting the frame rate helps make the game more playable this way, as does constraining the area of the screen that food can spawn in to only being a subsection at the center of the screen. It can actually require less movement than if the Snake’s head was tied to the nose position, but you still need to duck, sit up, and wobble your head a lot to play while seated.
I’m taking that as a lesson. Tying Snake to directly to the nose position probably wouldn’t have been very fun anyway. It could be fun, though, to use your hand/wrist while standing at a distance from the camera to do it, especially if using a better tracking solution. At that point we’re in the territory of reinventing the EyeToy/Kinect, I guess.–11/14/22