Linear algebra

Here is a very long and detailed note on linear algebra, formatted precisely according to the complex multi-column structure you provided.

---

Multicolumn

Blank

1. The Foundations: What is a Vector?

In linear algebra, we start with vectors. A vector is an object that has both magnitude (length) and direction. We can visualize them as arrows starting from an origin.

• In 2D: A vector $\vec{v}$ can be represented as coordinates: $\vec{v}=\left[\begin{array}{c}2\\ 3\end{array}\right]$ (2 units right, 3 units up).

• In 3D: $\vec{w}=\left[\begin{array}{c}1\\ 4\\ -2\end{array}\right]$ (1 unit on x-axis, 4 on y-axis, -2 on z-axis).

The collection of all possible 2D vectors is called ${\mathbb{R}}^2$. The collection of all 3D vectors is ${\mathbb{R}}^3$.

Transclude of 2D-Vector-Space.excalidraw

Multicolumn

Example

Vector Addition:

(Geometrically: tip-to-tail)

$\left[\begin{array}{c}1\\ 2\end{array}\right]+\left[\begin{array}{c}3\\ 1\end{array}\right]=\left[\begin{array}{c}1+3\\ 2+1\end{array}\right]=\left[\begin{array}{c}4\\ 3\end{array}\right]$

Example

Scalar Multiplication:

(Stretching or shrinking)

$5\cdot \left[\begin{array}{c}2\\ 1\end{array}\right]=\left[\begin{array}{c}5\cdot 2\\ 5\cdot 1\end{array}\right]=\left[\begin{array}{c}10\\ 5\end{array}\right]$

Blank

2. Organizing Data: What is a Matrix?

A matrix is a rectangular grid of numbers, arranged in rows and columns. If a vector represents a point, a matrix can represent a transformation of space itself.

• Dimensions: A matrix with $m$ rows and $n$ columns is called an "$m\times n$" matrix.

• Example (2x3 Matrix):

  $$A=\left[\begin{array}{ccc}1& 5& -2\\ 4& 0& 3\end{array}\right]$$

Matrices are the primary tool for representing systems of equations and linear transformations.

Multicolumn

Note

Notation: The entry in the $i$-th row and $j$-th column is written as $A_{ij}$. For matrix $A$ above, $A_{12}$ is 5.

Warning

Order Matters! Matrix multiplication (which we’ll see later) is generally not commutative. $A\times B\ne B\times A$.

---

Multicolumn

Blank

3. Linear Combinations & Span

This is a core concept. A linear combination is the result of adding vectors that have been scaled by scalars.

Given vectors ${\vec{v}}_1,{\vec{v}}_2,\dots ,{\vec{v}}_k$ and scalars $c_1,c_2,\dots ,c_k$, a linear combination is:

$\vec{y}=c_1{\vec{v}}_1+c_2{\vec{v}}_2+\cdots +c_k{\vec{v}}_k$

The span of a set of vectors $\{{\vec{v}}_1,\dots ,{\vec{v}}_k\}$ is the set of all possible linear combinations you can make from them.

Transclude of Span-of-two-vectors.excalidraw

Multicolumn

Info

Span in ${\mathbb{R}}^2$:

The span of a single vector (e.g., $\left[\begin{array}{c}1\\ 2\end{array}\right]$) is a line. The span of two non-parallel vectors (e.g., $\left[\begin{array}{c}1\\ 0\end{array}\right]$ and $\left[\begin{array}{c}0\\ 1\end{array}\right]$) is the entire 2D plane (${\mathbb{R}}^2$).

Info

Span in ${\mathbb{R}}^3$:

The span of two non-parallel vectors in ${\mathbb{R}}^3$ is a plane. To span all of ${\mathbb{R}}^3$, you need three vectors that don’t all lie on the same plane.

Blank

4. Systems of Linear Equations

Linear algebra is perfectly designed to solve systems of equations.

Example System:

$2x_1+3x_2=7$

$1x_1-1x_2=1$

We can write this in the compact matrix form $A\vec{x}=\vec{b}$:

$\underset{A}{\underbrace{\left[\begin{array}{cc}2& 3\\ 1& -1\end{array}\right]}}\underset{\vec{x}}{\underbrace{\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]}}=\underset{\vec{b}}{\underbrace{\left[\begin{array}{c}7\\ 1\end{array}\right]}}$

Multicolumn

Question

The Big Question:

Does a solution $\vec{x}$ exist? Is it unique? This is equivalent to asking: “Is the vector $\vec{b}$ in the span of the columns of matrix $A$?”

Tip

Geometric View:

Solving this system is finding the intersection point of two lines. The system could have one solution (lines intersect), no solutions (parallel lines), or infinite solutions (same line).

---

Multicolumn

Blank

5. Gaussian Elimination

This is the algorithm we use to solve systems of linear equations. The goal is to simplify the system into an “upper triangular” form that is easy to solve.

We use an augmented matrix and perform “elementary row operations”:

1. Swap two rows.

2. Multiply a row by a non-zero scalar.

3. Add a multiple of one row to another row.

Augmented Matrix for $A\vec{x}=\vec{b}$:

$\left[\begin{array}{ccc}2& 3& 7\\ 1& -1& 1\end{array}\right]$

We manipulate this to get Row-Echelon Form.

Transclude of Gaussian-Elimination-Steps.excalidraw

Multicolumn

Note

Reduced Row-Echelon Form (RREF):

A special form where all leading entries (pivots) are 1, and they are the only non-zero entries in their columns. This form directly gives you the solution.

Example

RREF for our system:

$\left[\begin{array}{ccc}1& 0& 2\\ 0& 1& 1\end{array}\right]$

This reads: $1x_1+0x_2=2$ and $0x_1+1x_2=1$.

The solution is $x_1=2,x_2=1$.

Blank

6. Linear Independence

This concept is related to “span” and “redundancy.” A set of vectors is linearly independent if no vector in the set can be written as a linear combination of the others.

• Informally: None of the vectors are “redundant.” Each vector adds a new direction to the span.

• Formally: The set $\{{\vec{v}}_1,\dots ,{\vec{v}}_k\}$ is linearly independent if the only solution to the equation

  $c_1{\vec{v}}_1+\cdots +c_k{\vec{v}}_k=\vec{0}$

  is the trivial solution: $c_1=0,\dots ,c_k=0$.

Multicolumn

Example

Independent:

$\left\{\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\right\}$. You can’t write one as a multiple of the other.

Example

Dependent:

$\left\{\left[\begin{array}{c}1\\ 1\end{array}\right],\left[\begin{array}{c}2\\ 2\end{array}\right]\right\}$. The second vector is just $2\times$ the first. It’s redundant. ${\vec{v}}_2-2{\vec{v}}_1=\vec{0}$.

---

Multicolumn

Blank

7. Basis & Dimension

A basis for a vector space (or subspace) is a “goldilocks” set of vectors. It has two properties:

1. It is linearly independent (no redundant vectors).

2. It spans the entire space (it has enough vectors to reach everywhere).

The dimension of a space is simply the number of vectors in its basis.

Multicolumn

Info

Standard Basis:

The most common basis for ${\mathbb{R}}^3$ is:

$\hat{ı}=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right],\hat{ȷ}=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right],\hat{k}=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$

Since there are 3 vectors, ${\mathbb{R}}^3$ is 3-dimensional.

Note

Uniqueness:

A space can have infinitely many different bases, but every basis for that space will always have the same number of vectors.

Blank

8. Linear Transformations

This is where matrices get their meaning. A linear transformation is a function $T$ that takes a vector as input and produces a vector as output, following two rules:

1. $T(\vec{u}+\vec{v})=T(\vec{u})+T(\vec{v})$ (Preserves addition)

2. $T(c\vec{v})=cT(\vec{v})$ (Preserves scalar multiplication)

Key Idea: A linear transformation is a matrix multiplication. Every $m\times n$ matrix $A$ corresponds to a linear transformation $T:{\mathbb{R}}^n\to {\mathbb{R}}^m$ defined by $T(\vec{x})=A\vec{x}$.

Transclude of Linear-Transformation-Grid.excalidraw

Multicolumn

Example

Rotation:

A 90° counter-clockwise rotation in ${\mathbb{R}}^2$ is a linear transformation.

Example

Projection:

Projecting all vectors in ${\mathbb{R}}^3$ onto the x-y plane is a linear transformation.

Warning

Not Linear:

$T(\vec{v})=\vec{v}+\left[\begin{array}{c}1\\ 2\end{array}\right]$ is not linear because it doesn’t map the zero vector to the zero vector ($T(\vec{0})\ne \vec{0}$).

---

Multicolumn

Blank

9. The Matrix of a Transformation

How do you find the matrix $A$ that corresponds to a specific linear transformation $T$?

The “Big Trick”:

You only need to see what $T$ does to the standard basis vectors.

1. Take the standard basis vectors: $\hat{ı}=\left[\begin{array}{c}1\\ 0\end{array}\right],\hat{ȷ}=\left[\begin{array}{c}0\\ 1\end{array}\right]$.

2. Apply the transformation to each one: $T(\hat{ı})$ and $T(\hat{ȷ})$.

3. The resulting vectors are the columns of your matrix $A$.

   $A=\left[\begin{array}{cc}|& |\\ T(\hat{ı})& T(\hat{ȷ})\\ |& |\end{array}\right]$

Transclude of Transformation-Matrix-Columns.excalidraw

Multicolumn

Example

Find the 90° Rotation Matrix:

1. $T(\hat{ı})=T(\left[\begin{array}{c}1\\ 0\end{array}\right])\to \left[\begin{array}{c}0\\ 1\end{array}\right]$ (1st col)

2. $T(\hat{ȷ})=T(\left[\begin{array}{c}0\\ 1\end{array}\right])\to \left[\begin{array}{c}-1\\ 0\end{array}\right]$ (2nd col)

3. The matrix is $A=\left[\begin{array}{cc}0& -1\\ 1& 0\end{array}\right]$.

Blank

10. Determinants & Inverses

The determinant is a single number, $\det (A)$, that tells you about the “scaling factor” of a transformation.

• If $\det (A)=2$, $A$ scales areas by a factor of 2.

• If $\det (A)=0.5$, $A$ shrinks areas by half.

• If $\det (A)=0$, $A$ “squishes” space into a lower dimension (like a line or a point).

• If $\det (A)<0$, $A$ “flips” the orientation of space (like a mirror reflection).

The inverse of a matrix, $A^{-1}$, “undoes” the transformation $A$.

$A{A}^{-1}=I$ (the identity matrix).

Multicolumn

Warning

Invertibility:

A matrix $A$ has an inverse $A^{-1}$ if and only if $\det (A)\ne 0$.

If the determinant is zero, the transformation squished space, and you can’t “un-squish” it (you lost information).

Example

Determinant (2x2):

$\det \left(\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\right)=ad-bc$

---

Multicolumn

Blank

11. Eigenvectors & Eigenvalues

This is the most important topic in linear algebra.

When a matrix $A$ transforms space, most vectors get “knocked” off their span (they change direction).

An eigenvector is a special, non-zero vector $\vec{v}$ that does not change its direction when transformed by $A$. It only gets scaled.

The scaling factor is its eigenvalue, $\lambda$.

The defining equation is:

$A\vec{v}=\lambda \vec{v}$

Transclude of Eigenvector-Transformation.excalidraw

Multicolumn

Info

What they mean:

Eigenvectors are the “axes of transformation.” They are the most “stable” directions in a system.

Example

Rotation: A 3D rotation’s eigenvector is its axis of rotation. Vectors on this axis don’t change direction (their eigenvalue is 1).

Blank

12. How to Find Eigenvalues

We need to solve $A\vec{v}=\lambda \vec{v}$.

1. Rewrite as: $A\vec{v}-\lambda \vec{v}=\vec{0}$

2. Insert Identity: $A\vec{v}-\lambda I\vec{v}=\vec{0}$

3. Factor out $\vec{v}$: $(A-\lambda I)\vec{v}=\vec{0}$

NOTE

bla bla car

We are looking for a non-zero vector $\vec{v}$ that solves this. This means the matrix $(A-\lambda I)$ must be non-invertible (it must “squish” $\vec{v}$ to zero).

And a matrix is non-invertible if its determinant is zero!

Characteristic Equation:

$\det (A-\lambda I)=0$

Multicolumn

Tip

The Process:

1. Solve the characteristic equation $\det (A-\lambda I)=0$ to find the eigenvalues $\lambda$.

2. For each $\lambda$, plug it back into $(A-\lambda I)\vec{v}=\vec{0}$ and solve for the eigenvectors $\vec{v}$.

Note

The set of all eigenvectors for a given $\lambda$ (plus the zero vector) forms a subspace called the eigenspace.