Quadratic form

Definition

A quadratic form turns a list of coordinates into one number using squares and products of pairs of coordinates. For example, $q(x_1,x_2)=2x_1^2+4x_1x_2+3x_2^2$ assigns the value $22$ to $(x_1,x_2)=(1,2)$. The matrix notation below stores the coefficients of such an expression compactly.

Notation to read first

A vector $x\in\mathbb R^n$ is a column of $n$ real numbers; its entries $x_1,\ldots,x_n$ are its coordinates. A scalar is a single number, such as $22$. A matrix $A\in\mathbb R^{n\times n}$ is an array with $n$ rows and $n$ columns; $a_{ij}$ is its entry in row $i$, column $j$. The main diagonal runs from the top-left entry to the bottom-right.

The transpose $A^T$ exchanges rows and columns. Thus $x^T$ turns a column into a row. For example,

The dot product multiplies matching coordinates and adds: $(1,2)^T\cdot(3,4)^T=1\cdot3+2\cdot4=11$. The identity matrix $I$ has diagonal entries $1$ and other entries $0$, so $Ix=x$. See the matrix-decomposition overview for these operations and OpenStax’s matrix notation for a basic introduction.

Formal definition

A real quadratic form is a function $q:\mathbb R^n\to\mathbb R$ (a vector goes in, one real number comes out) of the form

where $A\in\mathbb R^{n\times n}$ is fixed and $x\in\mathbb R^n$ is a column vector. The two summation signs mean “add $a_{ij}x_ix_j$ for every pair of indices $i,j$.” Every term has exactly two coordinate factors: $x_1^2$ counts as $x_1x_1$. There is no term like $5x_1$ (a linear term) or $7$ (a constant term). This is called homogeneous of degree two: scaling every coordinate by $t$ scales every term by $t^2$. For example, $2x_1^2+3x_2^2$ becomes four times as large when both coordinates double. See MIT’s notes, §5.2.

The dimensions explain why the result is a scalar:

Compute $Ax$ first, then take its dot product with $x$: $q(x)=x\cdot(Ax)$. The matrix maps a vector to another vector; the quadratic form assigns that vector a number. The Euclidean norm $\lVert x\rVert_2$ is the ordinary length $\sqrt{x_1^2+\cdots+x_n^2}$; $(3,4)^T$ has length $5$ and squared length $25$. See vector norms. When $A=I$, the quadratic form is this squared length $x^Tx=\lVert x\rVert_2^2$. More general matrices describe penalties, numbers measuring the cost of an input, and interactions between coordinates. Calling this value “energy” is an interpretation of that cost; the number need not represent physical energy.

The examples, derivations, and exercises below are original illustrations of these concepts.

Reading and constructing the matrix

The main diagonal runs from top left to bottom right; off-diagonal entries are the remaining positions. A matrix is symmetric if it equals its transpose, so entries reflected across the diagonal match. For example, the two off-diagonal entries below are both $b$. For such a two-by-two matrix,

A cross term multiplies different coordinates, such as $4x_1x_2$; its coefficient is the number $4$ multiplying that product. Compare $(x_1+x_2)^2=x_1^2+2x_1x_2+x_2^2$: the mixed product comes from multiplying the two different parts of the bracket. In $x^TAx$, the cross term occurs twice in the matrix product: once from $a_{12}x_1x_2$ and once from $a_{21}x_2x_1$. Therefore, to represent $3x_1^2+4x_1x_2+2x_2^2$ with a symmetric matrix, put half the cross-term coefficient in each off-diagonal position:

For $x=(1,-1)^T$, compute it both ways:

For three variables, the same rule gives

An absent interaction means a zero off-diagonal entry; it does not require the corresponding diagonal entry to vanish.

Why only the symmetric part matters

A skew-symmetric matrix satisfies $K^T=-K$: reflecting an entry across the diagonal reverses its sign, and the diagonal entries are zero. For example,

Those opposite contributions cancel. Any real square matrix can be split into a symmetric part $S$ and a skew-symmetric part $K$:

Because $K^T=-K$ and a scalar equals its transpose,

Thus $x^TKx=0$ and

This proves why restricting to symmetric representatives loses no quadratic forms; see MIT’s Exercise 84 and Theorem 86. A basis is a choice of building blocks from which every vector can be assembled uniquely. The blocks are independent: none can be built by scaling and adding the others. For example, the usual basis vectors $e_1=(1,0)^T$ and $e_2=(0,1)^T$ give $x=x_1e_1+x_2e_2$. Here $e_i$ means the vector with $1$ in position $i$ and $0$ elsewhere. The symmetric representative is unique for that coordinate choice, although adding any skew-symmetric matrix leaves the form unchanged.

For example,

both produce $2x_1^2+4x_1x_2+3x_2^2$. An eigenvector is a nonzero vector $v$ for which $Sv=\lambda v$: the matrix sends it to a multiple of itself. That multiplier $\lambda$ is its eigenvalue. For example, $\operatorname{diag}(2,3)$ means a matrix with $2,3$ on its main diagonal and zeros elsewhere; it sends $e_1$ to $2e_1$. Thus $e_1$ has eigenvalue $2$. See the overview’s eigenvector examples. To study the signs or geometry of our form, use the eigenvalues of $S$. Positive eigenvalues of a nonsymmetric $A$ alone do not suffice: \(A=\begin{bmatrix}1&4\\0&1\end{bmatrix}\) has only eigenvalue $1$, but $q(1,-1)=-2$.

Geometry: direction, scale, and level curves

For any real scalar $t$,

Doubling a displacement multiplies its value by four. Reversing its direction leaves the value unchanged. A linear function would satisfy both $q(tx)=tq(x)$ and $q(x+z)=q(x)+q(z)$. Quadratic forms generally fail these rules: for $q(x)=x_1^2$, $q(2e_1)=4$, whereas $2q(e_1)=2$.

Two vectors are orthogonal, or perpendicular, when their dot product is zero. A unit vector has length $1$. A square matrix $Q$ is orthogonal when its columns are perpendicular unit vectors; equivalently, $Q^TQ=I$. It rotates or reflects coordinates while preserving lengths. A diagonal matrix has zeros away from its main diagonal.

The spectral theorem says that a real symmetric matrix has a basis of perpendicular unit eigenvectors. Storing these vectors as the columns of $Q$ and their eigenvalues on the diagonal of $\Lambda$ gives $A=Q\Lambda Q^T$, where the columns of the orthogonal matrix $Q$ are unit eigenvectors and $\Lambda$ contains eigenvalues. Setting $y=Q^Tx$ gives

The eigenvector coordinates remove cross terms. See MIT’s diagonalization argument. Since $Q$ preserves lengths, each $\lambda_i$ measures the penalty along a unit eigenvector direction, also called a principal direction. Moving distance $t$ along it gives $q(tv_i)=\lambda_i t^2$. Its slope changes at rate $2\lambda_i$; this rate of change of slope is the second derivative of the one-variable parabola. This is what curvature means here; the derivatives section below explains the notation.

For a concrete change of axes, let \(A=\begin{bmatrix}2&1\\1&2\end{bmatrix}\). Its unit eigenvectors are $(1,1)^T/\sqrt2$ and $(1,-1)^T/\sqrt2$, with eigenvalues $3$ and $1$. Hence

A level curve, also called a contour, joins inputs with the same value: $q(x)=1$ means value $1$ at every point of that curve. Here $3y_1^2+y_2^2=1$ is an ellipse, a stretched circle. Its semiaxes are the distances from its center to its edge along its principal directions: setting $y_2=0$ gives $\lvert y_1\rvert=1/\sqrt3$, and setting $y_1=0$ gives $\lvert y_2\rvert=1$. A larger positive eigenvalue makes the same-value contour narrower in that direction.

The figure also previews the sign vocabulary: positive definite means positive for every nonzero input, positive semidefinite means never negative, and indefinite means positive in some directions and negative in others. For example, $x_1^2+x_2^2$, $x_1^2$, and $x_1^2-x_2^2$ have these three behaviors, respectively.

Read the figure as a map in the input plane, rather than a graph of height against position. If we instead draw the value as height, $x_1^2+x_2^2$ makes a bowl rising in every direction, $x_1^2$ makes a trough unchanged along $x_2$, and $x_1^2-x_2^2$ makes a saddle rising along one axis and falling along the other. A flat direction is one along which moving does not change the value; for this trough, adding any multiple of $(0,1)^T$ changes nothing.

Definiteness: what signs are possible?

Use the symmetric representative $S$ if the original $A$ is nonsymmetric. The abbreviations SPD and PSD mean symmetric positive definite and positive semidefinite, respectively. “Nonnegative” includes zero, whereas “positive” excludes it.

A negative definite form is strictly negative at every nonzero input; negative semidefinite allows zero but never positive values. These sign tests follow from the diagonal sum above; compare MIT’s Theorem 86. Definite forms are also semidefinite under the non-strict definitions; “semidefinite but not definite” specifically allows a nonzero zero-value direction. The zero form is both positive and negative semidefinite.

Positive diagonal entries do not prove positivity in every direction. For \(A=\begin{bmatrix}1&2\\2&1\end{bmatrix}\), both coordinate axes give positive values, but $q(1,-1)=-2$. Conversely, negative off-diagonal entries need not prevent positive definiteness:

The nullspace is the set of vectors sent to zero by a matrix: $Av=0$. For example, $\operatorname{diag}(1,0)$ sends $(0,1)^T$ to zero. A zero value at a nonzero vector need not mean a flat direction. For the indefinite form $x_1^2-x_2^2$, $q(1,1)=0$ even though $A(1,1)^T\ne0$. For a positive semidefinite symmetric matrix, however, $q(x)=0$ forces all positive-eigenvalue coordinates to vanish, so $Ax=0$.

Completing the square and Cholesky

Completing the square collects a square term and a cross term into a squared bracket, then keeps track of the remainder. For example, $t^2+2t+3=(t+1)^2+2$, because $(t+1)^2$ already supplies $t^2+2t+1$. In two coordinates, $(x_1+x_2)^2$ supplies the cross term $2x_1x_2$. See the Cholesky page’s square-completion explanation for more steps.

For $a>0$, direct expansion verifies

For \(A=\begin{bmatrix}4&2\\2&3\end{bmatrix}\), this becomes

This sum is strictly positive unless $x=0$. Set $y_1=2x_1+x_2$ and $y_2=\sqrt2x_2$; then $q(x)=y_1^2+y_2^2$. This change of coordinates is $y=L^Tx$, with

The factor $L$ is lower triangular, meaning its entries above the diagonal are zero. Its transpose is upper triangular: $y_2$ uses only $x_2$, whereas $y_1$ can use both old coordinates. That one-way dependence is what “triangular” means here.

Cholesky decomposition constructs such triangular sums of squares for symmetric positive-definite matrices. Its existence for SPD matrices is described by Higham. The remaining coefficient $c-b^2/a$ is a one-number Schur complement; for fixed $x_2$, it determines the minimum after allowing $x_1$ to adjust. Gallier, §2, develops the block version, grouping several rows and columns together rather than treating one coordinate at a time.

An orthogonal eigenvector transformation preserves lengths and reveals principal directions. The triangular Cholesky transformation instead rescales and mixes coordinates to make the penalty an ordinary squared length. These are two useful matrix decompositions with different purposes.

Derivatives and optimization

Optimization means choosing inputs to make a function as small or large as possible. The function we try to minimize is the objective. A global minimum is no larger than the function’s value at any other input.

A derivative measures slope. For several coordinates, a partial derivative changes one coordinate while holding the others fixed. The gradient $\nabla q$ collects these slopes into a vector. The Hessian $\nabla^2q$ is the matrix of second partial derivatives, describing how the slopes change. For example,

The zero off-diagonal entries say that changing $x_2$ does not affect the slope in $x_1$, or vice versa. See OpenStax’s partial and higher derivatives.

Let $h$ be a small change in the input. First-order change means the part proportional to the size of $h$; quadratic terms shrink faster as $h$ shrinks. For example, $(1+h)^2-1=2h+h^2$, whose first-order part is $2h$. For a quadratic form, expand $q(x+h)$ to see the first-order change:

The term quadratic in $h$ has no first-order contribution, so

For symmetric $A$, these reduce to $2Ax$ and $2A$. The common factor $1/2$ in an objective cancels that $2$:

Here $r$ is a fixed vector and $d$ a fixed scalar. $f$ is a quadratic function; its quadratic part is the form, while $-r^Tx$ is linear and $d$ is constant. A stationary point is an input where the gradient is zero: all coordinate slopes vanish. For example, $t^2-4t$ has slope $2t-4$, so $t=2$ is stationary. If $A$ is SPD, its stationary point $x_\star$ solves $Ax_\star=r$. Substituting this equation and expanding gives

which proves it is the unique global minimum. A stationary point of an indefinite quadratic need not minimize it. Convexity: slopes and curvature explains why a symmetric PSD Hessian gives a convex quadratic, how SPD strengthens this result, and why convexity alone does not guarantee an attained minimum.

Least squares chooses parameters $x$ to minimize the sum of squared differences between predictions $Bx$ and measurements $b$. Those differences form the residual $b-Bx$. Its squared length is the total squared error. For example, fitting one number $t$ to measurements $1,2,6$ minimizes $(1-t)^2+(2-t)^2+(6-t)^2$, with best fit $t=3$. In general,

Its quadratic form is $x^TB^TBx=\lVert Bx\rVert_2^2$, so it is nonnegative. Columns are linearly independent when none is a combination of the others. For example, $(1,0)^T$ and $(0,1)^T$ are independent, but $(1,0)^T$ and $(2,0)^T$ are not. Independent columns of $B$ make this form positive definite; otherwise some parameter changes cost nothing. See Higham’s Gram-matrix argument and the LU page’s normal-equation derivation.

Interactive example: change the coupling

The coupling here is the interaction recorded by the cross-term coefficient $b$. The energy explorer below uses \(A=\begin{bmatrix}a&b\\b&c\end{bmatrix}\) and reports $q(x)$ for your chosen vector. Keep $a=4$, $c=3$, and $x=(1,-1)^T$: predict $q(x)=7-2b$ before moving the coupling slider. Then compare the flat and indefinite presets and their displayed zero or negative directions.

The colors show signs and relative magnitudes on a fixed input range. A zero value at your selected vector alone does not classify the entire form. Without JavaScript, use the completed-square example above: $b=2$ gives positive definiteness, $b=\sqrt{12}$ gives a flat direction, and $b=4$ gives an indefinite form. These boundaries concern all directions, even while $q(1,-1)$ may remain positive.

Practice problems

These four original exercises cover hand calculation, interpretation, logical reasoning, and application. Attempt each before revealing the hint or solution.

1. Hand calculation: build and evaluate

Find the symmetric matrix for $q(x)=5x_1^2-6x_1x_2+2x_2^2$ and evaluate $q(1,2)$ by both matrix multiplication and substitution into the coordinate expression.

2. Interpretation: distinguish zero value from a flat direction

Compare $q_1(x)=(x_1+x_2)^2$ and $q_2(x)=x_1^2-x_2^2$. Both vanish at $v=(1,-1)^T$. Which is positive semidefinite? Which is indefinite? Does either satisfy $Av=0$?

3. Logic: recover a symmetric matrix from its form

Suppose symmetric matrices $S,T$ satisfy $x^TSx=x^TTx$ for every $x$. Prove $S=T$. Why does the conclusion fail without symmetry?

4. Application: minimize a coupled penalty

For \(A=\begin{bmatrix}4&2\\2&3\end{bmatrix}\) and $r=(6,5)^T$, minimize $f(x)=\frac12x^TAx-r^Tx$. Find the minimizer and prove it is global.

Further learning

• MIT 14.102, §5.2: Quadratic Forms: follow the transition from polynomial expressions to eigenvalue sign tests. The displayed negative-semidefinite inequality in Theorem 86 is $\lambda_i\le0$; its accompanying prose contains a sign typo.
• Higham: What Is a Symmetric Positive Definite Matrix?: connect positive quadratic values with Gram matrices and Cholesky factors.
• Gallier: The Schur Complement and Symmetric Positive Semidefinite (and Definite) Matrices: extend scalar square completion to coupled blocks of variables.

Related pages

• Symmetric positive definite matrix
• Cholesky decomposition
• Schur complement
• Matrix factorization and decomposition
• LU decomposition

References

1. Barczi, N. (2004). 14.102: Math for Economists, Lecture Notes, September 23. MIT, §5.2, Quadratic Forms.
2. Higham, N. J. (2020). What Is a Symmetric Positive Definite Matrix?.

3. Gallier, J. (2019). The Schur Complement and Symmetric Positive Semidefinite (and Definite) Matrices. University of Pennsylvania.

4. OpenStax. College Algebra 2e. Matrices and Matrix Operations. Matrix dimensions and multiplication.
5. OpenStax. Calculus Volume 3. Partial Derivatives. Slopes in separate coordinate directions and higher derivatives.