This note covers cameras: what the view and projection matrices actually do, and how to wire a mouse and keyboard to them.
There is no camera
The rendering pipeline has no such thing as a camera. Moving the camera really means moving the entire world the other way, so the observer stays at the origin looking down -Z.
That is the view matrix: the inverse of the camera's transform.
Swiftstruct Camera {
var position = SIMD3<Float>(0, 0, 5)
var rotation = SIMD3<Float>(0, 0, 0) // radians: pitch / yaw / roll
var viewMatrix: float4x4 {
let t = float4x4(translation: position)
let r = float4x4(rotation: rotation)
return (t * r).inverse
}
}
In practice, do not actually invert. A rigid transform has a closed-form inverse — transpose the rotation, negate the translation and transform it by that transposed rotation:
Swiftvar viewMatrix: float4x4 {
let r = float3x3(rotation: rotation).transpose
let t = -(r * position)
return float4x4(rotation: r, translation: t)
}
This is not only about speed; a general inverse is also numerically worse than the closed form.
The projection matrix and Metal's depth convention
The projection matrix maps the view frustum into clip space. Here lies a critical difference between Metal and OpenGL:
- OpenGL's NDC depth range is [-1, 1].
- Metal (like D3D and Vulkan) uses [0, 1].
Copy a projection matrix out of an OpenGL tutorial and in Metal you get "everything near the camera is clipped away", or depth testing that simply does not work. simd has no built-in projection matrix, so you have to write a correct one:
Swiftinit(perspectiveFov fov: Float, aspect: Float, near: Float, far: Float) {
let y = 1 / tan(fov * 0.5)
let x = y / aspect
let z = far / (far - near) // Metal: [0, 1], not (f+n)/(f-n)
self.init(
SIMD4<Float>( x, 0, 0, 0),
SIMD4<Float>( 0, y, 0, 0),
SIMD4<Float>( 0, 0, z, 1),
SIMD4<Float>( 0, 0, -z * near, 0)
)
}
The near plane matters far more than the far plane. Depth buffer precision is distributed uniformly in 1/z, which means almost all of it is spent close to the near plane. Moving near from 0.1 to 1.0 improves distant depth precision far more than dropping far from 10000 to 1000. The first response to z-fighting should always be push the near plane out.
The next step is reversed Z: map near to 1 and far to 0, with a floating-point depth buffer (.depth32Float) and a .greater compare function. Floats are most precise near 0, so reversing hands that precision to the distance, and depth accuracy improves by orders of magnitude. The cost is three changes: the projection matrix, the clear value (now 0), and the compare function.
Orthographic projection is for UI and shadow maps:
Swiftinit(orthographic rect: Rect, near: Float, far: Float) { ... }
It has no perspective divide, so parallel lines stay parallel and object size does not change with distance.
An orbit camera
A camera that rotates around a target point is the most convenient way to inspect a model:
Swiftstruct ArcballCamera {
var target = SIMD3<Float>.zero
var distance: Float = 5
var pitch: Float = 0 // radians
var yaw: Float = 0
var position: SIMD3<Float> {
let x = distance * cos(pitch) * sin(yaw)
let y = distance * sin(pitch)
let z = distance * cos(pitch) * cos(yaw)
return target + SIMD3(x, y, z)
}
var viewMatrix: float4x4 { .lookAt(eye: position, center: target, up: [0, 1, 0]) }
mutating func rotate(delta: CGPoint) {
yaw += Float(delta.x) * 0.01
pitch += Float(delta.y) * 0.01
pitch = min(max(pitch, -.pi / 2 + 0.001), .pi / 2 - 0.001) // avoid the flip
}
mutating func zoom(delta: Float) {
distance = max(0.5, distance - delta * 0.1)
}
}
Clamping pitch is essential. At exactly ±90° the view direction is parallel to the up vector, the cross product inside lookAt degenerates to a zero vector, and the camera snaps over. A small margin avoids it entirely.
Wiring up input
On macOS, an MTKView subclass can override the event methods directly:
Swiftclass InteractiveMTKView: MTKView {
var camera: ArcballCamera?
override var acceptsFirstResponder: Bool { true }
override func mouseDragged(with event: NSEvent) {
camera?.rotate(delta: CGPoint(x: event.deltaX, y: event.deltaY))
}
override func scrollWheel(with event: NSEvent) {
camera?.zoom(delta: Float(event.scrollingDeltaY))
}
}
For keyboard input on a first-person camera, do not move the camera inside keyDown. The key-repeat rate is decided by the system and has nothing to do with the frame rate, so movement speed becomes unpredictable. Keep a set of held keys and integrate against deltaTime once per frame in draw(in:):
Swiftprivate var keysDown = Set<UInt16>()
override func keyDown(with e: NSEvent) { keysDown.insert(e.keyCode) }
override func keyUp(with e: NSEvent) { keysDown.remove(e.keyCode) }
// once per frame
func update(deltaTime: Float) {
var move = SIMD3<Float>.zero
if keysDown.contains(13) { move.z -= 1 } // W
if keysDown.contains(1) { move.z += 1 } // S
if move != .zero {
camera.position += normalize(move) * speed * deltaTime
}
}
Do not skip the normalize(move) either: without it, holding W and D together moves you diagonally at √2 the speed — the first bug in essentially every first-person demo ever written.
From screen to world: picking
Turning a click into a world-space ray is the basis of object selection. Walk NDC coordinates back through the inverse of the projection:
Swiftfunc ray(fromScreen point: CGPoint, viewSize: CGSize) -> (origin: SIMD3<Float>, direction: SIMD3<Float>) {
let ndc = SIMD4<Float>(
Float(point.x / viewSize.width) * 2 - 1,
1 - Float(point.y / viewSize.height) * 2, // screen y is down, NDC y is up
0, // Metal's near plane is 0
1
)
let inv = (projectionMatrix * viewMatrix).inverse
var world = inv * ndc
world /= world.w // do not forget the perspective divide
let origin = camera.position
return (origin, normalize(world.xyz - origin))
}
Both of the easy mistakes are in the comments: screen y runs opposite to NDC y, and you must divide by w after transforming back.
The next note moves on to lighting.