DeepakBhalla

Matrix & Vector Algebra

Matrix notation was invented1 primarily to express linear algebra relations in compact form. Compactness enhances visualization and understanding of essentials.Note that all vectors and matrices are 1-indexed as standard

Indexes

  • $n$ - the number of distinct data points, or observations, in a sample.
  • $p$ - denote the number of variables that are available for use in making predictions
  • $X_{ij}$ - is an index reference which refers to the element in the ith row and $jth$ column of matrix X. where $i = 1,2,\cdots,n$ and $j = 1,2,\cdots,p$
  • $v_{i}$ is an index reference which refers to the element in the ith row of the vector $v$.
  • $i$ is be used to index the samples or observations (from 1 to n)
  • $j$ is be used to index the variables (from 1 to p)

Key Terms

Standard notations used across the board (unless specified elsewhere)

  • Matrices are usually denoted by uppercase names i.e. $X$
  • Vectors are lowercase bold i.e.$x$
  • 'Scalar' means that an object is a single value, and is not a vector or matrix.
  • Determinant is a useful value that can be computed from the elements of a square matrix.

Object Types

$a \in \mathbb{R}$ indicates that the object $a$ is scalar.

$a \in \mathbb{R}^n$ indicates that the object $a$ is a vector of length $n$.

$A \in \mathbb{R}^{n \cdot p}$ indicates that the object $a$ is a vector of columns $n$ and rows $p$.

Referencing

Matrices which have a single row are called row vectors, and those which have a single column are called column vectors

Let $X$ denote a $n×p$ matrix.

It can be indexed by its (i,j)th element as $x_{ij}$. Lower-case letters, with two subscript indices (for example, $x_{11}$, or $x_{1,1}$), represent the entries.

The entry in the i-th row and j-th column of a matrix A is sometimes referred to as the $i,j$, $(i,j)$, or $(i,j)$th entry of the matrix

An asterisk is occasionally used to refer to whole rows or columns in a matrix. For example, $x_{i,∗}$ refers to the $i$th row of X, and $a_{∗,j}$ refers to the $j$th column of A.

$$ X = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1p} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{np}\end {pmatrix} $$

A single row of $X$ can be referred to by $x_{1}, x_{2}, \cdots, x_{n}$ where $x_{1}$ can be written as this vector:

$$ x_{1} = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \vdots \\ x_{ip} \end{bmatrix} $$

A single column of $X$ can be referred to by $\boldsymbol{x_{1}}, \boldsymbol{x_{2}}, \cdots, \boldsymbol{x_{n}}$ where $\boldsymbol{x_{1}}$ can be written as this vector:

$$ \boldsymbol{x_{1}} = \begin{bmatrix} x_{1i} \\ x_{2i} \\ \vdots \\ x_{ni} \end{bmatrix} $$

Invertible Matrix

$A$ is invertible (also nonsingular or nondegenerate) if there exists an $n-by-n$ square matrix B such that.

$$ AB = BA = {I}_{n} $$

Where $I_n$ denotes the $n-by-n$ identity matrix and the multiplication used is ordinary matrix multiplication. $B$ would be called $A$'s inverse. The inverse of a matrix $A$ is denoted $A^{−1}$. Multiplying by the inverse results in the identity matrix.

Matrix Addition

$$ \begin{bmatrix}
a & b \\ c & d \\ \end{bmatrix} +\begin{bmatrix} w & x \\ y & z \\ \end{bmatrix} =\begin{bmatrix} a+w & b+x \\ c+y & d+z \\ \end{bmatrix} $$

Matrix-Scalar Multiplication

$$ \begin{bmatrix}
a & b \\ c & d \\ \end{bmatrix} \times x = \begin{bmatrix}
a \times x & b \times x \\ c \times x & d \times x \\ \end{bmatrix} $$

Matrix-Matrix Multiplication

Let $A \in \mathbb{R}^{n \cdot p}$ where $A = \begin{pmatrix} 1 & 3 & 5 \\ 2 & 4 & 6 \end{pmatrix}$

Let $B \in \mathbb{R}^{d \cdot s}$ where $B = \begin{pmatrix} 7 & 8 \\ 9 & 10 \\ 11 & 12 \end{pmatrix}$

Then matrix multiplication would produce the vector $AB$ of size $AB^{n \cdot s}$. Note matrix multiplication can only occur with two matrices where where the number of columns in A is the same as the number of rows B. Matrix mulitplication of $AB$ would work like this:

$$ AB = \begin{pmatrix} 1 & 3 & 5 \\ 2 & 4 & 6 \end{pmatrix} \begin{pmatrix} 7 & 8 \\ 9 & 10 \\ 11 & 12 \end{pmatrix} $$

$$ AB = \begin{pmatrix} (1 \times 7) + (3 \times 9) + (5 \times 11) = 89
\\ (2 \times 7) + (4 \times 9) + (6 \times 11) = 116 \end{pmatrix} $$ $$ append $$ $$ \begin{pmatrix} (1 \times 8) + (3 \times 10) + (5 \times 12) = 98
\\ (2 \times 8) + (4 \times 10) + (6 \times 12) = 128 \end{pmatrix} $$

$$ AB = \begin{pmatrix} 89 & 98 \\ 116 & 128 \end{pmatrix} $$