Lab M07 - Normal Vector Transformation

Overview

This worksheet explains why surface normals cannot be transformed with the model matrix when a non-uniform scale is present, derives the correct normal matrix, and shows when the shortcut of using the model matrix directly is safe. These concepts underpin every Phong and PBR lighting calculation in ACCB6014.

Learning Objectives

By the end of this lab you will be able to:

Demonstrate with a concrete example why applying the model matrix directly to a normal vector breaks perpendicularity under non-uniform scale
State the correct normal matrix formula and explain intuitively why it works
Compute the normal matrix for a given model matrix and write the C++ and GLSL code to use it
List the conditions under which the model matrix can be used directly as the normal matrix

Concept

When a model is scaled non-uniformly (differently along each axis), applying the same model matrix M to its surface normals rotates them to point in the wrong direction. The dot product between the wrong normal and the light direction gives the wrong angle. Lighting becomes visibly incorrect: surfaces that should appear brightly lit appear dark, or vice versa. This artefact is immediately noticeable in Phong shading and fatal in PBR.

A normal is not an ordinary direction vector. It encodes a perpendicularity constraint: for every surface tangent vector t, the normal n must satisfy n dot t = 0. When tangents are transformed by M, normals must be transformed by a different matrix N so that the constraint is preserved. The correct matrix is N = (M^-1)^T, the inverse-transpose of M. For orthonormal matrices (pure rotations), (M^-1)^T = M, so using the model matrix directly is safe. For non-uniform scales it is not.

In C++, compute the normal matrix on the CPU and pass it as a uniform: glm::mat3 normalMatrix = glm::mat3(glm::transpose(glm::inverse(model))). In the GLSL vertex shader, apply it with vec3 worldNormal = normalize(normalMatrix * aNormal). Always use the 3x3 block - normals are directions and directions ignore the translation column.

Tasks

Task 1 - The Failure Case

Setup: surface tangent t = (1, 1, 0) and normal n = (-1, 1, 0). These are perpendicular.

Verification:

n dot t = (-1)(1) + (1)(1) + (0)(0) = -1 + 1 + 0 = 0  ✓  perpendicular

Apply non-uniform scale M = scale(2, 1, 1):

M:
| 2  0  0  0 |
| 0  1  0  0 |
| 0  0  1  0 |
| 0  0  0  1 |

Transform the tangent:

Mt = M * (1, 1, 0, 0) = (2, 1, 0, 0)   (only the 3x3 block matters for directions)

Naively transform the normal with the same M:

Mn = M * (-1, 1, 0, 0) = (-2, 1, 0, 0)

Check perpendicularity:

Mt dot Mn = (2)(-2) + (1)(1) + (0)(0) = -4 + 1 = -3  ✗  NOT perpendicular

The naively-transformed normal no longer satisfies the perpendicularity constraint. Lighting computed with this normal will be wrong.

Compute the correct normal using the inverse-transpose:

M^-1 = scale(1/2, 1, 1):
| 1/2  0  0 |
|  0   1  0 |
|  0   0  1 |

(M^-1)^T = transpose of a diagonal matrix = the same diagonal matrix:
| 1/2  0  0 |
|  0   1  0 |
|  0   0  1 |

Apply (M^-1)^T to n = (-1, 1, 0):
correct_n = (1/2 * -1,  1 * 1,  1 * 0) = (-1/2, 1, 0)

Check perpendicularity with the correct normal:

Mt dot correct_n = (2)(-1/2) + (1)(1) + (0)(0) = -1 + 1 = 0  ✓  perpendicular

The correct normal should be renormalised before use in lighting calculations: normalise(-1/2, 1, 0) = (-0.447, 0.894, 0).

The original n dot t = , confirming they are perpendicular.

After naive transform: Mt dot Mn = , confirming perpendicularity is broken.

The correct normal matrix is written as (M)^T.

After applying the correct normal matrix: Mt dot correct_n = .

Task 2 - Why the Inverse-Transpose Works

The following is an intuitive argument, not a formal proof. The goal is to understand why the formula N = (M^-1)^T preserves the perpendicularity constraint.

We need a matrix N such that (Mt) dot (Nn) = 0 whenever t dot n = 0.

Rewrite the dot product as a matrix product: (Mt)^T(Nn) = 0, which expands to t^T M^T N n = 0.

For this to hold whenever t^T n = 0, we need M^T N = I (the identity). Solving for N:

M^T * N = I
N = (M^T)^-1 = (M^-1)^T

In plain language: the normal matrix is the matrix that undoes what M does to the perpendicularity relationship. The result N = transpose(inverse(M)) is exactly the operation needed.

When does this simplify?

If M is a pure rotation: M^-1 = M^T, so N = (M^T)^T = M. Using the model matrix directly is safe for pure rotations.

If M is a uniform scale S*I: N = (1/S)*I, which only changes the magnitude of n. Renormalising the normal after applying M fixes this, so the model matrix (or its 3x3 block) is also safe after renormalisation.

For a pure rotation matrix R, what is the inverse-transpose (R^-1)^T?

A. The zero matrix B. R itself (the rotation matrix unchanged) C. The negative of R D. R transposed twice, which is undefined

For a rotation matrix, R^-1 = R^T. Therefore (R^-1)^T = (R^T)^T = R. The normal matrix equals the model matrix for pure rotations, so the shortcut is safe.

Task 3 - Compute the Normal Matrix

Matrix A: M = scale(3, 3, 3) - a uniform scale.

Compute (M^-1)^T for this matrix.
Is the shortcut (using M directly as the normal matrix) safe here?

Matrix B: M = scale(2, 1, 1) composed with an arbitrary rotation R. Do not compute the product by hand.

Is the shortcut safe for matrix B?
Write the C++ GLM code to compute the correct normal matrix for model matrix B.
Write the GLSL vertex shader line that applies the normal matrix to the vertex normal.

Matrix A = scale(3,3,3). Compute (M^-1)^T. Is the model matrix shortcut safe? Explain.

M^-1 = scale(1/3, 1/3, 1/3). (M^-1)^T = scale(1/3, 1/3, 1/3) (diagonal matrix: transpose equals itself). The shortcut IS safe here. Uniform scale only changes the magnitude of the normal, not its direction. After renormalising (which lighting code always does), the result is identical to using the inverse-transpose. Using M or (M^-1)^T both give correctly oriented normals after normalisation.

Matrix B = scale(2,1,1) * R. Is the shortcut safe? Write the C++ GLM code for the correct normal matrix and the GLSL line to apply it.

The shortcut is NOT safe. M contains a non-uniform scale (2 along X, 1 along Y and Z). Non-uniform scale distorts normal directions in a way that renormalisation cannot fix. C++ (compute on CPU, pass as uniform): glm::mat3 normalMatrix = glm::mat3(glm::transpose(glm::inverse(model))); GLSL vertex shader: vec3 worldNormal = normalize(normalMatrix * aNormal); Note: computing inverse() in the vertex shader itself is expensive and should be avoided. Always compute the normal matrix on the CPU and pass it as a uniform.

Task 4 - When Is the Shortcut Safe?

List two conditions under which you can use the model matrix (or its 3x3 block) directly as the normal matrix, without computing the inverse-transpose. Explain why each condition is sufficient.

1. The model matrix contains only rotations (and optionally a uniform scale). Reason: for pure rotations, M^-1 = M^T, so (M^-1)^T = M. The normal matrix equals the model matrix. A uniform scale factor cancels out after normalisation, so it does not affect the final result. 2. The model matrix is the identity. Reason: trivially safe. The identity transforms nothing. I^-1 = I and I^T = I, so (I^-1)^T = I = the model matrix. In any other case (non-uniform scale, shear), the full inverse-transpose must be computed.

Solution

Try the tasks above before revealing the solution.