Vector norms

Observation errors: why one number is needed

In his 1805 appendix on least squares, Legendre asks how to determine unknown quantities from observations whose equations cannot generally all be satisfied. His original symbolic model is $E=a+bx+cy+fz+\cdots$: the coefficients are known, the unknowns are shared between equations, and $E$ is an error, the discrepancy left in an equation. He chooses to minimize the sum of squared errors. See the translated original text in Smith’s A Source Book in Mathematics, p. 576 and p. 577. This is a motivating problem for Euclidean error measurement, not a claim that Legendre introduced today’s norm axioms.

Derive the size measure from the original problem

A vector is an ordered list of coordinates. We write it as a column; the superscript $T$ transposes a row list into that column. The notation $\mathbb R^n$ means vectors with $n$ real coordinates, and $\lvert x_i\rvert$ is the absolute value of coordinate $i$.

Original symbolic problem, modern re-derivation. Legendre also considers repeated measurements of one quantity and the position of a point (pp. 578–579). Write the measurements as $a_1,\ldots,a_n$ and the proposed common value as $t$. The errors are $e_i(t)=a_i-t$. The observations are real values of the same quantity, expressed in a common unit and given equal weight. Legendre’s unweighted objective is

Let $\bar a=(a_1+\cdots+a_n)/n$ be their arithmetic mean. Because $\sum_i(a_i-\bar a)=0$, expanding the squares gives

The second term is nonnegative and vanishes only at $t=\bar a$. This proves the mean minimizes the objective, without needing differential calculus. The vector $e=(e_1,\ldots,e_n)^T$ collects all discrepancies; $\sqrt{S(t)}$ is its Euclidean length. Minimizing $S$ or $\sqrt S$ selects the same $t$, since the square root preserves order for nonnegative numbers.

For position measurements, Legendre sums the squared coordinate differences in ordinary spatial distance. Modern vector notation writes each such distance as $\lVert u-v\rVert_2$. The translation verifies these symbolic problems but omits the ensuing meridian application and its numerical data. We do not reconstruct missing observations. The numerical worked example below instead preserves a later published example of measuring computational error; no physical units are specified there.

Published worked data: LAPACK’s error vector

Published illustrative data, modern recalculation. The first vector example after Table 4.2 in the 1999 LAPACK Users’ Guide, “How to Measure Errors” gives

Here $x$ is the reference vector and the hat marks an approximation. Subtract coordinate by coordinate:

Three possible summaries now answer different questions:

The sign tells which way a coordinate misses; it does not cancel the magnitude. Relative error compares a discrepancy with the size of a reference. Dividing the maximum error by the maximum reference coordinate gives $2/100=0.02$. Dividing the total absolute error by the total absolute reference size gives $3.1/110\approx0.0281818$.

Denominator convention matters. LAPACK also reports ratios using the approximate vector as denominator: $2/99\approx0.0202020$ and $3.1/111.1\approx0.0279028$. These are the guide’s rounded $0.0202$ and $0.0279$, not the ratios with $x$ in the denominator. The lab shows both conventions. Global relative error can hide a bad small coordinate: the third coordinate misses by $2/9\approx22.2\%$ even though the maximum-norm global ratio is $2\%$.

Primary interactive example: replay the published error data

The reference vector stays fixed. Change the three approximate coordinates, predict which error summary will change, then restore the published baseline. All edits are hypothetical experiments; the defaults are the guide’s illustrative numbers, not measured observations. The worked calculation above is the no-JavaScript version of the baseline.

Generalize: the norm axioms

A vector has several coordinates, but a norm assigns one nonnegative number to its size. The observation problem suggests Euclidean length; the computational example shows why total and maximum magnitude can also be useful. The following axioms specify which measuring rules behave consistently under scaling and addition.

For vectors $x,y\in\mathbb R^n$, a norm is a function $\lVert\cdot\rVert:\mathbb R^n\to\mathbb R$ satisfying:

1. Positive definiteness: $\lVert x\rVert\ge0$, with equality exactly when $x=0$.
2. Absolute homogeneity: $\lVert\alpha x\rVert=\lvert \alpha\rvert\lVert x\rVert$ for every real scalar (number used as a multiplier) $\alpha$.
3. Triangle inequality: $\lVert x+y\rVert\le\lVert x\rVert+\lVert y\rVert$.

The same axioms apply over $\mathbb C$, using the complex modulus. See Higham’s vector-norm definition. Homogeneity says doubling a vector doubles its size; the triangle inequality says combining two displacements cannot cost more than taking them separately.

A norm induces distance by $d(x,y)=\lVert x-y\rVert$. This measures the difference between two vectors rather than their individual sizes.

Common norms

For $x=(x_1,\ldots,x_n)^T$ and $1\le p<\infty$,

These are the standard $p$-norms; the infinity norm is their limit as $p\to\infty$. See Higham and LAPACK’s norm table.

Supplementary worked example: one vector, three questions

This two-dimensional example, the geometric figures, and the practice exercises are supplementary invented illustrations. They are not Legendre’s observations or LAPACK’s published data.

For $x=(3,-4)^T$:

The negative sign describes direction; it does not cancel size. Traveling along horizontal and vertical streets costs $7$ units, traveling straight costs $5$, and the largest coordinate displacement is $4$.

To normalize in a chosen norm, divide a nonzero vector by its norm. Here $x/5=(0.6,-0.8)^T$ has Euclidean norm $1$, but its $1$-norm is $1.4$. “Unit vector” needs a specified norm. The zero vector cannot be normalized.

Geometry: the unit ball changes shape

The unit ball contains vectors with $\lVert x\rVert\le1$; its boundary has $\lVert x\rVert=1$. In two real dimensions:

• $\lvert x_1\rvert+\lvert x_2\rvert\le1$ is a diamond.
• $x_1^2+x_2^2\le1$ is a circular disk.
• $\max(\lvert x_1\rvert,\lvert x_2\rvert)\le1$ is an axis-aligned square.

For $(1,1)^T$, the three sizes are $2$, $\sqrt2$, and $1$. Thus a bound on the infinity norm allows several coordinates to be large at once. A bound on the $1$-norm instead gives coordinates a shared total budget.

Supplementary interactive example: change the vector and the measuring rule

The defaults $(3,-4)$ are invented teaching values. The restore button returns to those defaults, and edits are hypothetical. Predict the sizes at $(1,1)$, $(3,-4)$, and $(0,0)$. Change the selected norm and watch the normalized point move to the new boundary. The line marks the input vector, while the boundary records the selected definition of size. Inputs and the selector work with a keyboard. Without JavaScript, the calculations and static figure above show the same comparison.

Comparing norms without recalculating everything

Let $M=\max_i\lvert x_i\rvert$. One term contributes $M^2$ to the sum of squares, and every term is at most $M^2$, so

Also, expanding $(\sum_i\lvert x_i\rvert)^2$ adds nonnegative cross terms to $\sum_i\lvert x_i\rvert^2$, while the Cauchy–Schwarz inequality, $\lvert \sum_i u_iv_i\rvert\le\sqrt{\sum_i u_i^2}\sqrt{\sum_i v_i^2}$, bounds the absolute sum of coordinate products by the product of Euclidean lengths. For example, $u=(1,1)$ and $v=(3,4)$ give $7\le\sqrt2\cdot5$. See ALAFF, Theorem 1.2.3.3. Applying it to $(\lvert x_i\rvert)$ and $(1,\ldots,1)$ gives

The constants matter: $x=(1,\ldots,1)^T$ has sizes $n$, $\sqrt n$, and $1$. In fixed finite dimension these bounds make convergence in any of these three norms equivalent, but numerical thresholds differ. If $\lVert e\rVert_\infty\le0.01$ for $100$ error coordinates, then $\lVert e\rVert_2\le0.1$; one cannot infer $\lVert e\rVert_2\le0.01$.

Choosing an error measure

For a computed vector $\hat x$, absolute error is $\lVert\hat x-x\rVert$ and relative error is $\lVert\hat x-x\rVert/\lVert x\rVert$ when $x\ne0$. See LAPACK’s error-measure definitions. Specify the norm with the tolerance.

For errors $e=(0.1,-0.1,0.1,-0.1)^T$, the maximum error is $0.1$, the Euclidean error is $0.2$, and the total absolute error is $0.4$. A per-coordinate tolerance naturally uses the infinity norm; a total squared-error objective uses $\lVert e\rVert_2^2$. If coordinates mix units or have different scales, rescale them deliberately before combining them. A meter and a second do not acquire a shared meaning simply by appearing in the same array.

Common misconceptions and counterexamples

Squared Euclidean length is not a norm. For $x=(1,0)^T$, $\lVert2x\rVert_2^2=4$, whereas homogeneity would require $2\lVert x\rVert_2^2=2$. It is a useful objective, but it scales quadratically.

The “zero norm” is not a norm. Counting nonzero coordinates is often denoted $\lVert x\rVert_0$. Scaling $(1,0)^T$ by $2$ leaves the count at $1$, violating homogeneity. NumPy also supports such quantities while explicitly distinguishing them from mathematical norms.

The $p$ formula needs $p\ge1$. At $p=1/2$, take $x=(1,0)^T$ and $y=(0,1)^T$. The formula gives sizes $1$ and $1$, but size $4$ for $x+y$, violating the triangle inequality.

Not every norm is rotation invariant. A $45^\circ$ rotation takes $(1,0)^T$ to $(1/\sqrt2,1/\sqrt2)^T$. Euclidean size stays $1$; the $1$-norm becomes $\sqrt2$. Coordinate-based error rules depend on the axes chosen.

Practice problems

1. Hand calculation: compute and normalize

Find the $1$, $2$, and infinity norms of $x=(-2,1,2)^T$, then normalize it in the Euclidean norm.

2. Interpretation: concentrated or spread-out error?

Compare $e=(4,0,0,0)^T$ and $f=(1,1,1,1)^T$. Which has smaller total absolute error? Which has smaller maximum error? What does this tell you about selecting a tolerance?

3. Abstract reasoning: a weighted size

For positive weights $w_i$, prove $N(x)=\sum_i w_i\lvert x_i\rvert$ is a norm. What fails if one weight is zero?

4. Application: validate a displacement estimate

An exact displacement is $x=(3,4)^T$ meters and an estimate is $\hat x=(3.03,3.96)^T$ meters. Find the absolute and relative Euclidean errors. Does a maximum coordinate-error tolerance of $0.035$ meters pass?

Further learning

• Legendre’s translated appendix, pp. 576–579: follow the original observation-error problem and the mean and position derivations. The introduction is later editorial context; the translated appendix is the original argument.
• Higham: What Is a Vector Norm?: continue from the norm axioms to dual norms and bounds on inner products.
• LAPACK: How to Measure Errors: connect norms to absolute error, relative error, and conditioning.
• NumPy: numpy.linalg.norm: see how array shape, ord, and axis select the mathematical quantity being computed.

Related pages

• Matrix norms
• Symmetric positive definite matrix
• Matrix factorization and decomposition

References

1. Higham, N. J. (2021, September 22). What Is a Vector Norm?. Norm axioms and the $p$-norm family.
2. Anderson, E., et al. (1999). LAPACK Users’ Guide, third edition. How to Measure Errors. Standard norms and absolute and relative error.

3. NumPy developers. numpy.linalg.norm. Software conventions and quantities that do not satisfy the norm axioms.

4. Legendre, A.-M. (1805). “Sur la méthode des moindres quarrés,” Nouvelles méthodes pour la détermination des orbites des comètes, pp. 72–75. English translation by H. A. Ruger and H. M. Walker in Smith, D. E. (1929), A Source Book in Mathematics, pp. 576–579. Original symbolic error model and repeated-measurement problem; the excerpt omits subsequent numerical meridian data.
5. van de Geijn, R. A., and Myers, M. D. Advanced Linear Algebra: Foundations to Frontiers, “The vector 2-norm (Euclidean length),” Theorem 1.2.3.3. Cauchy–Schwarz inequality used in the norm comparison.