Even after taking a college level signal processing course, I didn't really grok why we use sines and cosines. Any explanation of Fourier theory just starts with the idea to break up a signal into sine and cosine components. Perhaps they'd say those are the simplest, purest instantiation of a frequency. But the idea still just falls out of the sky.
I believe I know now: it's because of the convolution theorem. When performing a Fourier transform, we represent the signal in a new basis, where each component will independently get transformed by linear shift invariant systems.
Basically the Fourier basis diagonalizes the convolution operator. And the even deeper reason for that is that a complex exponential function can be shifted by simply multiplying it with a constant.
In simpler words, it comes down to the barber pole illusion. If you rotate a spring-shaped 3d curve, it looks as if it was traveling upwards. And the 2d projection of the spring are sines and cosines.
And it turns out that linear, shift-invariant systems are really common (or are at least a good approximation of many common natural phenomena), so it's very helpful to break up a signal into pieces that each get independently transformed, without any interaction effects.
It's also easier to motivate them (and get an intuition for them) if you have a physical motivation for them. Like if you imagine being in Fourier's shoes (or Heaviside's) and needing to come up with techniques for solving differential equations more easily. Fourier series let you solve PDEs with linear algebra, and Laplace transforms turn ODEs into polynomial equations, both of which are easier to deal with.
Or to put it another way, if you imagine being faced with diff eqs and thinking "Hmm, is there a way I can turn these into problems I already know how to solve... like, say, polynomials?", then the idea that there even exists an invertible transformation that lets you do such a thing is itself quite fascinating in its own right—regardless of any elegance the transformation itself might have from a theoretical mathematical standpoint. And having such a motivation and application in mind (and knowing the physical nature of the problem, where we know we'll end up with waves) helps ground the idea in something very concrete and avoids making it look like it's just a random transformation we're studying because we're bored.
> I didn't really grok why we use sines and cosines.
We use sines and cosines, but not only sines and cosines. There are many other interesting sets of functions. For example, polynomials, or derivatives of the gaussian functions (called Hermite functions).
Bonus point: sines and cosines are polynomials evaluated on the complex unit circle.
> Basically the Fourier basis diagonalizes the convolution operator.
It also diagonalizes the second derivative, which is the linear operator that governs many physical processes, like wave propagation and heat diffusion.
If you evaluate the complex function f(x)=z^n on the unit circle z=exp(it), you obtain cos(nt)+i sin(nt). Thus Fourier series are "just" Taylor series evaluated on the unit circle.
> I didn't really grok why we use sines and cosines.
As has been pointed out elsewhere in this thread, they have nice mathematical properties. But another important thing is that they typically work "well enough" for applications. Consider audio. Tones clearly have frequency, but they also have a position in time. Doing a sine-cosine decomposition of a whole song doesn't really make sense, since it has no way of saying that a tone on the piano is played at a given time.
So you would think that it would make sense to break the signal down into stuff with frequency and time. Some kind of wavelet probably. Maybe something that very accurately models what a human hears.
The thing is that chopping the audio stream up into windows, and decomposing those windows into sines and cosines, while a bit ad hoc, just works well enough.
> So you would think that it would make sense to break the signal down into stuff with frequency and time. Some kind of wavelet probably. Maybe something that very accurately models what a human hears.
Interestingly wavelet based compression went nowhere because although they have nice mathematical properties, when applied in a lossy compression scheme, they did not fit well with how humans perceive detail/quality, both in terms of psychoaccoustics and psychovisuals, i.e. PSNR vs subjective quality diverged more than with other systems. Not surprisingly none of the state of the art lossy compression algorithms use wavelets.
Yes, that didn't help of course. JPEG2000 was a bit more efficient than JPEG by virtue of being a newer, more computationally intensive format, not because of wavelets. A modern format like the h.265 derived HEIF still uses DCT/DST based transforms. For video the situation is worse, as AFAIK no-one has been able to come up with a decent wavelet based motion estimation algorithm.
Sine (or cosine) is just the projection of uniform circular motion onto a line segment.
Why we break arbitrary periodic functions into a sum of sines and cosines (or circular motion of different integer frequencies) is because uniform circular motion is very well understood and studied, so we have many tools for working with it. It’s very easy and convenient to isolate particular terms, and each term has a pretty simple shape, and is infinitely differentiable. These bases are conveniently orthogonal relative to a uniform weight.
It's because the eigenvectors of the collection of all time shift operators are the exponential functions, and if you want to stick with real functions, then sin(ax) and cos(ax) together form a basis for the sum of the ai and -ai eigenspaces. (In the non-periodic case, you also have to deal with e^(cx) and products of this with sin and cos for all the "transients"). If you are doing something that is time-invariant, that means it commutes with time shift operators, and that means it has the same eigenvectors. Importantly, this implies such an operator can be analyzed (more or less) by how it scales these eigenvectors.
> Perhaps they'd say those are the simplest, purest instantiation of a frequency.
Actually I thought it is the other way round: A frequency is defined as a sine basically, it has an
* amplitude
* frequency
* phase shift
The phase shift is why we need cosines and sines in the Fourier transform as cosine is just a shifted sine.
The frequency is the constant in the argument of the sine. And here is where the tail might chase the dog: it's the definition, not an observation I'd think.
> When performing a Fourier transform, we represent the signal in a new basis, where each component
... each component is a sine, that is described by the 3 constants above.
> In simpler words, it comes down to the barber pole illusion. If you rotate a spring-shaped 3d curve, it looks as if it was traveling upwards. And the 2d projection of the spring are Sines and cosine.
Exactly, a complex exponential function is just that spring. And if the absolute value of the argument is not exactly one, the spring spirals outwards or inwards.
> Actually I thought it is the other way round: A frequency is defined as a sine basically, it has an * amplitude * frequency * phase shift
But this is also true of any periodic wave. The beginner question is, why don't we use triangle waves of square waves? Those also have amplitude, frequency and phase. Frequency just means the reciprocal of the period.
To which my answer is that the magical property of complex exponential functions is that they can be shifted by constant (pointwise) multiplication. Which is a really non-obvious fact at first but is crucial in the machinery.
The complex exponentials constitute an orthogonal basis which diagonalizes the convolution.
I think it's that exponentials simultaneously diagonalize time shifts, derivatives, and integrals. Convolution follows from these -- though there's something to be said for putting convolution above derivatives and integrals in importance.
(A deeper thing going on is that a periodic function can be thought of as a function whose domain is a circle. Circles have obvious rotational symmetry, and when you have symmetry you can use representation theory to decompose things into an (orthogonal) basis. In this case, rotations commute with each other so by some theory the decomposition is going to be entirely through eigenvectors of rotation, which happen to precisely be the exponential functions e^(n theta i) for n an integer. This decomposition is also an isomorphism that carries convolutions to point-wise products in both directions. Also: if you make it so the circle is the complex unit circle, a Fourier transform is the idea that you can create a Laurent polynomial that extends the function to the complex plane minus the origin.)
Nice, indeed at the deeper level it's that if you want to make time shifts equivalent to rotation, you need to move in a circular fashion.
Minor point: to me derivatives are just one specific linear time invariant operator, a kind of convolution (with a generalized function) so I think LTI is the thing we really care about.
Does the representation theoretic perspective / harmonic analysis also explain why the Laplace transform works? I would be interested to know if it's possible to pick a different compact group from S1 to recover the Laplace transform
I've been thinking about this since my original comment actually, but there are complexities from using the noncompact group R. The exponential functions are still the simultaneous eigenvectors of time shifts.
The Laplace transform seems to be just using the fact that <f,g> = integrate(f(x) g(x), x from 0 to infinity) is an inner product for the space of square integrable functions (probably better would be <f,g> = integrate(f(x) conj(g(x)), x from 0 to infinity) as a Hermitian product). The various e^(ax) functions are linearly independent, so the functions g |-> <e^(ax), g> are linearly independent functionals. If the exponential functions are actually enough, then this means you can study a function by studying the vector consisting of its value through all the functionals, which is the Laplace transform.
The Laplace transform has a pretty bad inverse formula, partly because the exponential functions are not orthogonal with respect to the inner product.
Sines and cosines really are fundamental. Pass a triangular, sawtooth or square wave through a narrow-band bandpass filter (an actual physical filter) centred on the fundamental frequency and you'll get a sine wave out the other end. Anything that isn't sinusoidal has components that aren't at the fundamental frequency.
> The beginner question is, why don't we use triangle waves of square waves
You can think in terms of complex exponentials, and the use of complex numbers makes even more sense when you know the differential equations can be solved by polynomial methods, which naturally leads to complex numbers.
However you can also answer "why sine waves not triangle/square waves" with a geometrical answer. (This is how I learned it at school, before I knew about complex numbers or differential equations.)
A property of sine waves is that their sums and products are also sine waves or simple combinations of a small number of sine waves.
The sine wave shape persists. This neat property is unique to sine waves, and you can think of it as a type of symmetry.
For example, in the simplest cases: adding two sine waves with the same frequency and different phase produces another sine wave with that same frequency. Multiplying two sine waves with different frequencies produces the same as a sum of two sine waves, having the sum-of-frequencies and difference-of-frequencies.
Doing so with any other wave shape results in a different wave shape than you started with.
A lot of audio and radio signal processing depends on this property, and the way sum-of-frequencies and difference-of-frequencies works for every sine wave component at the same time. Our whole concept of a radio "band" of signals that you can tune into comes from it.
It's about FFT but it goes through the (less known) polynomial multiplication foundation which seems more practical and related to the convolutional applications.
Here are some properties that make the cosine base popular. It has the property that among a large set of bases, it roughly provides the best approximation of a given function [0]. Moreover, this approximation can be efficiently obtained by evaluating the function on Chebyshev nodes [1], followed by computing a discrete cosine transform [2].
A neat bit of history is that Fourier analysis was originally developed by Fourier to solve a problem - the heat equation - which has no waves in the solution, just diffusion!
Sinusoids have nice mathematical properties, but there's no law of nature that says you have to use them. Walsh transforms use square waves (actually binary rather than purely square) as basis functions and they have other handy mathematical properties.
There is no why. The cos and sin form a basis of your function pace. If it aligns with what you want to achieve you use sin and cos, if not you use a different basis (special case of this is e.g. the wavelet transformation)
One reason for sines and cosines is that they are the solutions for a particular differential equation that seems to show up all over the place in physics and engineering.
Fourier transform is so fundamental to understanding the world and mastering Fourier transform (and linear algebra too) is so sooo important before entering anything "quantum". Quantum field theory (QFT), especially, is essentially a way to look at the world through its Fourier transform.
In a nutshell Fourier transform, the derivative operator, exponential function and CAUSALITY are related at a very deep level. And that's why it's such an important mathematical tool. Because it is related to causality.
I understand all the noted mathematics. Any suggestions for references on learning quantum theories? For most materials, I get stuck as they use notations unfamiliar to me in spite of me being familiar with the underlying mathematics.
You could greatly benefit from Prof. Leonard Susskind's lectures on quantum mechanics and QFT on Youtube (Standford university's channel), which are very accessible. A great physicist who happens to be a great teacher and who gets all the deep interconnections involved. This is one playlist (he has many more): https://www.youtube.com/watch?v=JzhlfbWBuQ8&list=PL84C10A9CB...
Fun fact: the human auditory system works by splitting incoming sounds into frequencies. Each hair has a different characteristic resonant mode, so our brain processes information in f-space.
Fourier is legit magic. A physical manifestation of Fourier is light being split into it's component frequencies when passing through a prism. I think that this can be used to solve NP in P. All current thinking happens in the context of Turing machines, however if you can factorize a number in O(1) (not that a prism has any concept of Big O) things are different.
> We're going to leave the mathematics and equations out of it for now. There's a bunch of interesting maths behind it, but it's better to start with what it actually does, and why you'd want to use it first. If you want to know more about the how, there's some further reading suggestions below!
Yes, yes, yes, yes! Anybody teaching really has to understand that if you need a formula to explain a concept, you haven't understood the concept yourself OR you simply don't know how to explain it yet.
Sure, there are exceptions (there always are), but in my experience, lecturers, teachers and academics reach for formulas to "explain" things too quickly. Formulas are not explanations. They serve as PROOF that what you explained actually works.
You can use FFT principles to isolate and remove repeated patterns in images (such as the patterns created by those textured papers of old photographs).
Wow this is very cool! Do you know how people achieve something that works like FFT but without much latency? Go example an EQ that you can draw any shape of.
I believe I know now: it's because of the convolution theorem. When performing a Fourier transform, we represent the signal in a new basis, where each component will independently get transformed by linear shift invariant systems.
Basically the Fourier basis diagonalizes the convolution operator. And the even deeper reason for that is that a complex exponential function can be shifted by simply multiplying it with a constant.
In simpler words, it comes down to the barber pole illusion. If you rotate a spring-shaped 3d curve, it looks as if it was traveling upwards. And the 2d projection of the spring are sines and cosines.
And it turns out that linear, shift-invariant systems are really common (or are at least a good approximation of many common natural phenomena), so it's very helpful to break up a signal into pieces that each get independently transformed, without any interaction effects.