Linear Algebra

I: What Exactly Is a Vector

The most basic, most fundamental building block of linear algebra is the vector. Generally speaking, there are three ways of looking at a vector:

1. From a physicist's perspective: a vector is an arrow in space, defined by its length and the direction it points. A vector in a plane is two-dimensional, while a vector in the space we live in is three-dimensional. As long as the length and direction stay the same, we're free to move it around without the vector changing.
2. From a computer scientist's perspective: a vector is an ordered list of numbers. For instance, suppose we're doing some analysis related to housing prices, and we only care about two features: house area and price. We'd model a house using a pair of numbers:

The first is the house area, and the second field is the price. Note that the order of these numbers cannot be swapped. We'd model the house using a two-dimensional vector.

3. Mathematicians try to generalize these two viewpoints: broadly speaking, a vector can be anything, as long as vector addition and scalar multiplication of a vector are well defined.

In short, vector addition and scalar multiplication run throughout linear algebra and play a very important role.

II: Linear Transformations

A transformation refers to the process of taking an input and producing an output. Any given transformation can be arbitrarily complex. A linear transformation refers to a special class of transformations.

Geometrically speaking, if a line remains a straight line after the transformation — without any curving — and the origin stays fixed, then it's a linear transformation.

In short, this means the grid lines remain parallel and evenly spaced. Some linear transformations are easy to picture, such as rotation around the origin, scaling, and shearing.

Next, let's describe linear transformations numerically. Given a vector, applying a linear transformation to it produces another vector.

It turns out that we only need to record where and land after the transformation. Every other vector will follow accordingly.

III: Matrix-Vector Multiplication

A linear transformation is completely determined by its effect on the space's basis vectors, because any other vector can be expressed as a linear combination of the basis vectors.

The fact that grid lines remain parallel and evenly spaced after a linear transformation leads to a beautiful consequence:

As long as we record where the basis vectors land after the linear transformation, we can determine where any vector lands after the transformation. It's simply vector x times , plus vector y times .

For example, suppose we have a vector , which is equivalent to . We then apply a linear transformation such that . The transformed vector is then

In other words, this turns vector-matrix multiplication into scalar multiplication and addition of vectors.

Here, we call the vector , and the vector .

This can therefore be extended to multiplication between an n-dimensional vector and a matrix:

Where:

Multiplying a matrix by a vector is precisely applying the linear transformation to that vector.

IV: Matrix Multiplication

Often, we want to describe an effect like this: after applying one transformation, apply another. For example, we might want to describe rotating first, then shearing. In general, this is a composite linear transformation.

How should we describe such a composite transformation? This is what matrix multiplication does:

Note that matrix multiplication is applied from right to left. We can understand this as: and first undergo the transformation on the right, then the transformation on the left.

and first move to the position , undergoing vector-matrix multiplication. Then the vector undergoes the same multiplication, and the final result is the matrix product.

We can extend this to the n-dimensional case:

Suppose we have two transformations:

The matrix product is:

Where:

V: Affine Transformations

Geometrically, a linear transformation has three key properties:

• Lines before the transformation remain lines after the transformation
• Ratios along a line remain unchanged
• The origin before the transformation remains the origin after the transformation

Numerically, a linear transformation is implemented via matrix multiplication.

An affine transformation, geometrically, has only two key properties:

• Lines before the transformation remain lines after the transformation
• Ratios along a line remain unchanged

It's missing the requirement that the origin stays fixed. Translation, for example, is an affine transformation rather than a linear transformation, because the origin moves.

So numerically, an affine transformation requires not just matrix multiplication, but also addition.

For the formula above, we can raise the dimensionality to turn the translation into a linear transformation.

After adding a dimension, we can accomplish the lower-dimensional affine transformation through a linear transformation in the higher dimension.

Geometrically, this can be understood as adding a axis, so that translation becomes a transformation along the direction — thereby ensuring that the grid lines remain parallel and evenly spaced.