{"id":1847,"date":"2024-02-07T16:17:10","date_gmt":"2024-02-07T16:17:10","guid":{"rendered":"https:\/\/timallanwheeler.com\/blog\/?p=1847"},"modified":"2024-07-20T03:21:47","modified_gmt":"2024-07-20T03:21:47","slug":"posing-meshes-with-opengl","status":"publish","type":"post","link":"https:\/\/timallanwheeler.com\/blog\/2024\/02\/07\/posing-meshes-with-opengl\/","title":{"rendered":"Posing Meshes with OpenGL"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

This month I’m continuing the work on the side scroller and moving from static meshes to meshes with animations. This is a fairly sizeable jump as it involves a whole host of new concepts, including bones and armatures, vertex painting, and interpolation between keyframes. For reference, the Skeletal Animation post on learnopengl.com<\/a> prints out to about 16 pages. That’s a lot to learn. Writing my own post about it helps me make sure I understand the fundamentals well enough to explain them to someone else.<\/p>\n\n\n\n

Moving Meshes<\/h1>\n\n\n\n

Our goal is to get our 3D meshes to move. This primarily means adding having the ability to make our player mesh move in prescribed ways, such as jumping, crouching, attacking, and climbing ladders. <\/p>\n\n\n\n

So far we’ve been working with meshes, i.e. vertex and face data that forms our player character. We want those vertices to move around. Our animations will specify how those vertices move around. That is, where the \\(m\\) mesh vertices are as a function of time:<\/p>\n\n\n\n

\\[\\vec{p}_i(t) \\text{ for } i \\text{ in } 1:m\\]\n\n\n\n

Unfortunately, a given mesh has a lot of vertices. The low-poly knight model I’m using has more than 1.5k of them, and its relatively tiny. Specifying where each individual vertex goes over time would be an excessive amount of work.<\/p>\n\n\n\n

Not only would specifying an animation on a per-vertex level be a lot of work, it would be very slow. One of the primary advantages of a graphics card is that we can ship the data over at setup time and not have to send it all over again:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

With the rendering we’re doing now, we just update a few small uniforms every frame to change the player position. The mesh data itself is already on the GPU, stored as a vertex buffer object.<\/p>\n\n\n\n

So fully specifying the vertex positions over time is both extremely tedious and computationally expensive. What do we do instead?<\/p>\n\n\n\n

When a character moves, many of their vertices typically move together. For example, if my character punches, we expect all of the vertices in their clenched fist to together, their forearm arm to follow. Their upper arm follows that, maybe with a bend, etc. Vertices thus tend to be grouped, and we can leverage the grouping for increased efficiency.<\/p>\n\n\n\n

Bones<\/h1>\n\n\n\n

You know how artists sometimes have these wooden figures that they can bend and twist into poses? That’s the industry-standard way to do 3D animation.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

We create an armature<\/em>, which is a sort of rigid skeleton that we can use to pose the mesh. The armature is made up of rigid entities, bones<\/em>, which can move around to produce the movement expected in our animation. There are far fewer bones than vertices. We then compute our vertex positions based on the bones positions – a vertex in the character’s fist is going to move with the fist bone.<\/p>\n\n\n\n

This approach solves our problems. The bones are much easier to pose and thus build animations out of, and we can continue to send the mesh and necessary bone information to the GPU once at startup, and just send the updated bone poses whenever we render the mesh.<\/p>\n\n\n\n

The bones in an armature have a hierarchical structure. Every bone has a single parent and any number of children, except the root bone, which has no parent.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

Unlike my oh-so-fancy drawing, bones don’t actually take up space. They are actually just little reference frames. Each bone’s reference frame is given by a transformation \\(T\\) with respect to its parent. More specifically, \\(T\\) transforms a point in the bone’s frame to that of its parent.<\/p>\n\n\n\n

For example, we can use this transform to get the “location” of a bone. We can get the position of the root bone by transforming the origin of its frame to its parent – the model frame. This is given by \\(T_\\text{root} \\vec{o}\\), where \\(\\vec{o}\\) is the origin in homogenous coordinates<\/a>: \\([0,0,0,1]\\). Our transforms use 4×4 matrices so that we can get translation, which is ubiquitous in 3D graphics. <\/p>\n\n\n\n

Similarly, the position of one of the root bone’s children with transform \\(T_\\text{c}\\) can be obtained by first computing its position in the root bone’s frame, \\(T_\\text{c} \\vec{o}\\), and then convert that to the model frame, \\(T_\\text{root} T_\\text{c} \\vec{o}\\). <\/p>\n\n\n\n

The order of operation matters! It is super important. It is what gives us the property that moving a leaf bone only affects that one bone, but moving a bone higher up in the hierarchy affects all child bones. If you ever can’t remember which order it is, and you can’t just quickly test it out in code, try composing two transforms together. <\/p>\n\n\n\n

The origin of a bone 4 levels deep is:<\/p>\n\n\n\n

\\[T_\\text{root} T_\\text{c1} T_\\text{c2} T_\\text{c3} T_\\text{c4} \\vec{o}\\]\n\n\n\n

Let’s call the aggregated transform for bone \\(c4\\), this product of its ancestors, \\(\\mathbb{T}_{c4}\\).<\/p>\n\n\n\n

It is these transformations that we’ll be changing in order to produce animations. Before we get to that, we have to figure out how the vertices move with the bones.<\/p>\n\n\n\n

Transforming a Vertex<\/h1>\n\n\n\n

The vertices are all defined in our mesh. They have positions in model-space. <\/p>\n\n\n\n

Each bone has a position (and orientation) in model space, as we’ve already seen. <\/p>\n\n\n\n

Let’s consider what happens when we associate a vertex with a bone. That is, we “connect” the vertex to the bone such that, if the bone moves, the vertex moves with it. We expect the vertex to move along in the bone’s local frame:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

Here the bone both translates to the right and up, and rotates counter-clockwise about 30 degrees. In the image, the vertex does the same.<\/p>\n\n\n\n

The image above has the bone starting off at the origin, with its axis aligned with the coordinate axes. This means the vertex is starting off in the bone’s reference frame. To get the vertex into the bone frame (it is originally defined in the model frame), we have to multiply it by \\(\\mathbb{T}^{-1}\\). Intuitively, if \\(\\mathbb{T} \\vec{p}\\) takes a point from bone space to model space then the inverse takes a point from model space to bone space.<\/p>\n\n\n\n

The bone moved. That means it has a new transform relative to its parent, \\(S\\). Plus, that parent bone may have moved, and so forth. We have to compute a new aggregate bone transform \\(\\mathbb{S}\\). The updated position of the vertex in model space is thus obtained by transforming it from its original model space into bone space with \\(\\mathbb{T}^{-1}\\) and then transforming it back to model space according to the new armature configuration with \\(\\mathbb{S}\\):<\/p>\n\n\n\n

\\[\\vec v’ = \\mathbb{S} \\mathbb{T}^{-1} \\vec v\\]\n\n\n\n

We can visualize this as moving a vertex in the original mesh pose to the bone-relative space, and then transforming it back to model-space based on the new armature pose:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

This means we should store \\(\\mathbb{T}^{-1}\\) for each bone in the armature – we’re going to need it a lot. In any given frame we’ll just have to compute \\(\\mathbb{S}\\) by walking down the armature. We then compute \\(\\mathbb{S} \\mathbb{T}^{-1}\\) and pass that to our shader to properly pose the model.<\/p>\n\n\n\n

Bone Weights<\/h1>\n\n\n\n

The previous section let us move a vertex associated with a single bone. That works okay for very blocky models composed of rigid segments. For example, a stocky robot or simple car with spinning wheels. Most models are less rigid. Yes, if you punch someone you want the vertices in your fist to follow the hand bone, but as you extend your elbow the vertices near the joint will follow both the bone from the upper arm and the bone from the lower arm. We want a way to allow this to happen.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

Instead of a vertex being associated with one bone, we allow a vertex to be associated with multiple bones. We have the final vertex combination be a mix of any number of other bones:<\/p>\n\n\n\n

\\[\\vec v’ = \\sum_{i=1}^m w^{(i)} \\mathbb{S}_i \\mathbb{T}_i^{-1} \\vec v\\]\n\n\n\n

where the nonnegative weights \\(\\vec w\\) sum to one.<\/p>\n\n\n\n

In practice, most vertices are associated with one one or two bones. It is common to allow up to 4 bone associations, simply because that covers most needs and then we can use the 4-dimensional vector types supported by OpenGL.<\/p>\n\n\n\n

Data-wise, what this means is:<\/p>\n\n\n\n