As an electrical-engineer turned machine-learning-grad-student, linear transformations have been involved in most everything I do since my first year of undergrad. But all this time, I've done matrix multiplication the way I was taught in high school: "The (i,j) element of AB is what you get by walking right across the i'th row of A while you walk down the j'th column of B, taking the sum of products as you go."
It works, but there's no connection between that process and the intuition of a linear transformation; it's just a rote computation. And checking a long string of matrix multiplications to see if they intuitively make sense (shouldn't everything intuitively make sense?) is especially aggravating when you constantly have to interrupt your intuition to switch to a rote calculation.
I never thought to think of the columns of B as vectors that physically travel through A; to think of a dataflow or pipeline from right to left on the page. Sure, it's not a cure-all, but it'll be a useful mental tool to have.
Oh, and it's also an excellent introduction to the subject, although the Linear Operations section gets a bit muddled... first something's not a linear operation, and then it is, wat? Still, an excellent post.
You might get some mileage out of thinking of a row in the matrix product as a dot product, with the intuition that goes along with that. The dot product is perhaps easier to tie into the geometric intuition you have for linear transformations.
Can't say enough good things about Prof. Gilbert Strang http://www-math.mit.edu/~gs/ He is one of my heros, and yeah, I have copied his Linear Algebra lectures to CD so I can speed them up, stop them, and in case they ever disappear from the net. Not to take away from the OP, who I think has some good ideas for making this stuff intuitive.
I'm watching them now, and they are simply superb, probably the best math class I've ever "taken". I particularly liked his "4 ways of matrix multiplication" (lecture 3) and explanation of fourier series (lecture 24), but his exposition of every concept from determinants to eigenvalues has been intuitive and memorable.
> Linear algebra gives you mini-spreadsheets for your math equations.
Okay, that lets you visualize it (in the finite case) but it's a terrible way to sell it. Spreadsheets are booooring. Did you know that functions are vectors? Okay, better. Did you know that quantum mechanics is all about linear algebra? Okay, sold!
1. Almost any time you work in more than one dimension you will want linear algebra in your toolbox. There are a zillion methods for solving (non-linear) equations out there, and in more than one dimension, they use linear algebra. Newton's method? Incredibly useful in practice due (quadratic convergence rocks!), and with some linear algebra sauce BOOM you have Newton's method in as many dimensions as you can sneeze at.
2. Oh, by the way... did you know that the Fourier transform is linear?
3. Back to quantum mechanics... there's a thing you can do with a linear operator (a matrix is a kind of linear operator) where you get the "spectrum" of the linear operator. It's useful for making sense of big matrices. But in QM, the wavefunctions for electrons are described as eigenfunctions of a linear operator, and taking the "spectrum" of the linear operator gives you the actual spectrum of light that the chemical under study emits. Hence the name, "spectral theorem". It may be linear algebra on paper, but it's laser beams and semiconductors in the real world.
4. Oh hey, want to learn about infinite-dimensional vector spaces? Maybe some other time..
5. It's hella useful for modeling. Any model is wrong, but Markov processes are useful. Say you run an agency that rents out moving vans, and you have facilities in 30 cities. Vans rented in city A have a 10% chance of being dropped off in city B, 7% in city C, 9.2% in city D, etc. At this rate, how long till you run out of vans in city F? It's a differential equations problem with like 30 different equations! Or you could rewrite it as a single equation with matrices. You'll end up with weird things like 'e^(A*t)' where A is a matrix, and you thought "no way I can exponentiate to the power of a matrix" but spectral decomposition is like "yes way!" and you can solve the equation by diagonalization. Radical! (Basically, linear algebra rescues differential equations from the pits of intractability. I'm using rental vans as an example, but it could be a chemical reaction or a nuclear reaction or a million subway riders or whatever you want.)
So the question is:
Do you find economics, quantum physics, chemistry, engineering, classical mechanics, machine learning, statistics, etc. useful?
Hate to do this (but will have to anyway because you're misrepresenting that authors article),
- you've mixed in with nonlinear phenomena (most physical processes) with linear, and there's less and less reason to pretend linearity with increasing computer speeds (although in fairness you alluded to models being wrong)
- poo pooed spreadsheets without explaining why (you alluded to QM etc. but they don't care about useless proofs in linear algebra unless they're pencil pushing time wasters)
- implied infinite-dimensions is practically useful when it isn't (unless you're a mathematician/theoretical-somethingist seeking to extract tax-payer money)
There's little need to get linear algebra in you unless you want to waste time.
You're underlying point throughout this discussion seems to be that you think there are folks who overestimate the importance of mathematics (especially the abstract parts of it), those people are often smug, and sometimes more prone to pontificating than productivity.
Those people may exist. But I don't see any of them here.
You've gone too far in the other direction by saying it's a stupid and complete waste of time to study math. There are real breakthroughs in insight there, even if sometimes obscured by pompous nonsense.
Your comments are fueled by anger, rather than a sincere effort to inform others, and this is why you've been downvoted.
I'm willing to bet you can be a constructive contributor here if you try. You're probably right that a simplified and more pragmatic approach to learning math might make it accessible to a larger group of people. If so, it would be better to go help make that happen than to blindly criticize everyone else.
I'm not angry at all. How can I add to the anti-math movement, I thought this might be a good venue, there's lots of thoughtful and productive people here. A lot of them might not be aware of the societal devastation caused by the current maths system (making most people not believe in their intellectual abilities and consequent destruction in wealth creation). Math anxiety news sometimes comes up and people like the Wolfram brothers make a case but on the whole people, including geeks, haven't realized how much computers have displaced mathematics.
You seem to have had a traumatic experience trying and failing to learn some mathematics. That sucks for you. Lots of people unfortunately go through that, but you seem to have developed a unique coping reaction. Rather than lay the blame on the school system, the curriculum, your specific teachers, or (perish the thought) partly on yourself, you have decided that mathematics is a vast conspiracy to keep lazy pinheads employed and make everyone else feel dumb and inferior. Your hostile way of demanding evidence for its utility (without defining what would constitute acceptable evidence) is a foolproof algorithm for rejecting anything whatsoever. I've seen hardheaded engineers deny the value of the field of physics using exactly the kind of arguments you've laid out in this thread. They seem to believe that every aspect of physics with engineering applications could just as easily have been discovered with common sense and simple-minded trial and error.
Not at all, I never tried getting education in it in the first place. I had hunch it's useless and the older I get the more I think that was correct, although I do look at it for practical utility and never find any (I'm not arguing from a position of not knowing anything). People, like on this thread, bring up stuff and it's comes across as religious dogma, they don't explain how in their daily work a specific example of where it worked (and they should include how there were no other alternatives).
There's a huge, huge range of frauds being committed in society. I think people should point them out when they see it.
edit: I also agree mostly with the hard-headed engineers you brought up. Physics is somewhat useful, but it's also vastly overtaught and overfunded relative to it's practical utility. I'm an AI guy and think we'll get the singularity before any of the fundamental research going on pays off (and that's only a small proportion of the world's wealth being spent on physics education for people who will never find practical use for it). And I think there's also a lot of mathemagic symbol throwing in there as well to extract tax-payer money (a lot of it caused by the mathemagicians's influence on physicists).
If you are genuinely interested in examples of its practical utility, I suggest you lay out clear criteria for what would constitute acceptable evidence to you. It might also help to give examples of which non-trivial parts of mathematics you do find useful, with reasons for why.
I looked at his posting history first before making an effort to communicate. He seems to have an extreme, hostile bias against theory but otherwise comes across as well-intentioned, if misguided.
Hate to do this, but it seems you just don't understand jack about linear algebra. I'm not even sure you read what I wrote.
1. Yes, I mixed in linear and non-linear processes. My point is that you use linear algebra even when working with non-linear processes.
2. Yes, I poo-pooed spreadsheets as a bad way to market linear algebra (I thought that was clear?)
3. Yes, I implied infinite dimensions is useful. Linear algebra in QM tends to be infinite-dimensional.
I'm reading a lot of hostility towards mathematics in general and linear algebra specifically (you've put a bunch of toxic comments all over this thread), and I'm not really sure why.
> Yes, I implied infinite dimensions is useful. Linear algebra in QM tends to be infinite-dimensional.
You don't have to appeal to QM for their usefulness. Function spaces are important in applied and computational mathematics because of their use in understanding integral equations (the theory of integral equations before Hilbert spaces was a giant mess), partial differential equations, calculus of variations, approximation theory, etc. No doubt marshallp will respond that this is all bullshit because it ultimately boils to finite processes running on finite state machines, where there are no infinite sets in sight, let alone infinite dimensions. The equivalent approach to physics would be an extreme form of empiricism, banning the use of concepts like electrons (as some logical positivists actually proposed to do in the early twentieth century) and requiring all physical laws to be stated in terms of directly observable phenomena, whatever that means.
I hope that this matches how some people think enough to help them.
For me it is too computational. I prefer understanding the topic from first principles as described at http://news.ycombinator.com/item?id=4086325. (Then again I don't particularly like spreadsheets either.)
After going through that course I finally understood things like eigenvectors, null spaces, and projections. Now I see them everywhere (unless you think that's a curse)
Funny, I'm actually working on a game that intends to teach some of this stuff to a non-mathematical audience.
Linear algebra is so far-reaching, I find it surprising that other branches of mathematics often seem to get preferential treatment (usually normal algebra and geometry), in spite of the fact that linear algebra is both:
a) fairly advanced (i.e. not often taught in school, at least not the deeper stuff)
b) not very difficult to learn (unlike lots of other 'introductory' topics in mathematics).
Perhaps there is something about matrices (being mere tables of numbers for most folks) that people find unattractive, almost statistics-like.
(On the other hand, it could be a simple extension of the symbol barrier [1], given those long vertical brackets.)
"The eigenvectors are the axes of the transformation" = mind blown. After several engineering courses, studying eigenvectors in an advanced math class and still no one could put it this simple. This guy is amazing.
His description of the determinant, too. I hadn't heard that explanation until a second semester of real analysis, when learning the proof of the Inverse Function Theorem (an amazing thing to study, by the way, connects many the dots between linear algebra and calculus).
Even then, it was a question I had to ask my brilliant, constantly-pissed looking young professor. "Hey, uh, the Jacobian... what does the determinant mean, uh, geometrically?". He looked at me like a slug, before explaining it was the measure of the newly mapped unit square. Fireworks went off in my head. Two linear algebra classes before only ever explained it by its algorithm or its usefulness (e.g., ∃ A^(-1) for A \in R^{n,n} iff det(A) != 0)
Side note: that professor had the most effective teaching style for pure math I've ever seen. Besides lectures that expanded on the contents of Rudin and interesting problem sets, he gave us a list of a hundred theorems, propositions, and exercises. Told us the final exam would be six problems, four of which would come from that list, another of which would be a clever new one, the last something truly hard.
Never learned analysis better than when sitting down and working through (not memorizing) each of those proofs and theorems for possible later recapitulation.
Determinant as volume of the transformation is cool, but I still don't "get" the determinant intuitively despite many years of math. In particular, why does the determinant work to solve linear equations (i.e. Cramer's rule)? And what's the motivation behind the formula for the determinant? (I realize these questions are a bit vague, but I'm hoping for a more intuitive answer than "that's just the way the math works out".)
reg. "And what's the motivation behind the formula for the determinant?", if I'm understanding your question right, the neat little pic at http://en.wikipedia.org/wiki/Determinant#2-by-2_matrices - might be what you were looking for.
I wish that I'd had a similar experience. I bought Halmos "Linear Algebra Problem Book" based on the accolades, but the lack of motivation/inspiration caught me offguard and left me sorely disappointed before the end of chapter 1. If you can appreciate maths without pictures, you have to be either truly gifted or totally deluded.
Recently, learning to do Principal Components Analysis to solve a handwriting recognition problem is what finally shed an enchanted light on linear algebra. The way that a simple matrix of data samples is transformed into an ordered set of principle components (eigenvectors, ordered by eigenvalue) is.. "unreasonably effective" (as they say). The principal components are your signal, and the rest (with eigenvalues ~0) are the noise. the handwriting recognition, btw, works fantastic for my simple application. no need for non-linear kernels and whatnot.
Did you see that on Jeremy Kun's great blog? His primers are how I recently got into building my own Entropy-trained decision tree class and also got an intuition for PCA as a reduced basis. I had used truncated SVD and Fourier bases many times before, but to see it with images (eigenfaces!) really sold the intuition.
Even better now, in this hacking life after pure math in college and grad school, is that I can build intuition now not just by proofs and exercises, but also efficient, coded implementation. Gives a different feel for the tools and concepts.
What the author uses as his strawman is a linear algebra course for engineers.
As mathematicians, we didn't do any of this matrix and vector stuff with numbers when introducing linear algebra in university. There were a bunch of axioms, and you proved things. That's how you know.
What the author sees as abstract "(2d vectors! 3d vectors!)" was way more applied than the stuff we dealt in.
But, granted, the purpose wasn't learning about how to get mini-spreadsheets for equations. It was about how to rigorously navigate a useful axiomatic setting.
(Later on, we proved that you can find a base, and write down your linear transformation as a bunch of numbers and call that a matrix; same for points and vectors. But we always saw that as somewhat ugly, and anyway limited to the finite dimensional case.)
I had studied it under a similar approach. Of course, after taking Abstract Algebra course, and seeing Polynomials over Abstract Fields, which form a vector space, while computing the degree of finite field extensions[1], I realized how much powerful Linear Algebra was.
Also during a Computational Complexity course, when we were studying de-randomization techniques, computing the Spectrum of an expander graph[2] and looking into the eigenvalues and eigenvectors of the graphs' adjacency matrix to understand topological properties from graphs, I realized how powerful and general Linear Algebra can be.
2d and 3d vectors over the Field of Real Numbers are easy to see, and a good way to get started. But many beautiful and powerful things come from this theory.
Linear Algebra was at the core of almost every math and cs course I took. I ended up taking several semesters of it and by the end we were approaching highly relevant problems in subjects like quantum mechanics and machine learning. I cannot recommend studying it highly enough!
If anyone is interested in building an understanding of Linear Algebra from the ground up, I would suggest some of Israel Gelfand's work: http://www.scribd.com/doc/35985818/Gelfand-Lectures-on-Linea...
It's not for the light hearted, but when you get through it, you will be deeply changed and have a new perspective on even basic math, such as integration.
> What the author uses as his strawman is a linear algebra course for engineers.
If that's true, I'm sorry for the engineers who took this course.
I did not see a single number in my algebra course, neither 2d or 3d vectors. As a matter of fact, we followed the same program that math students (axioms, theorems, linear spaces, rings, ...). Even some teachers were teaching in both faculties, and our exams usually were more difficult (this was just a policy, there were too many engineering students).
Now, I'm in a different university where the maths future engineers study are much more "practical". It's sad seeing that students cannot understand, for example, that a principal direction in a stress tensor is just an eigen-value or that a rotation is just a change of base and, therefore, there are many theorems and properties you can apply to these concepts just making use of some basic algebra.
It's always good to know well the tools you have to use and, for an engineer, algebra is one of the most important tools.
If you can't prove a theorem, then it's not a theorem, it's a conjecture. Conjectures are not as useful. For example, the Shannon-Hartley theorem tells us under what circumstances it is possible to decode a radio signal. That theorem lets us design, e.g., cell phone networks.
Theorems are useful in all sorts of unexpected ways. At the turn of the 20th century group theory was considered a worthless corner of mathematics, but understanding group theory allows modern chemical instrumentation to work.
edit:
> It seems like it's the full employment act for pencil pushers
> downvoters downvote instead of providing refutation. A lot in common with fundamental religionists
You've proven the downvoters right by piling one ad hominem attack upon another.
The example you've described (radio signals) is just a method of winning an argument. It doesn't actually help you, you practically do it by writing a computer program that tries out different parameters to get the right one.
Let's say you need a communications channel with a 300 Mbit/s capacity. Shannon's theorem lets you know what kind of parameters make that possible -- namely, bandwidth and signal-to-noise ratio. From there, I can make an informed decision about the entire signal chain. Once I choose the bandwidth, the theorem tells me the SNR so I can assign a noise budget to each of the components. If we assume that the noise is Gaussian, we can calculate total noise level as the root sum of squares of the noise levels of the individual components (don't forget to multiply by the gain). I can look at an amplifier IC's spec sheet and immediately say, "that's too noisy" or "that's overdesigned, I think I can get something cheaper".
And thanks to theorems we got from the field of real analysis and statistics, we know that the sum of Gaussians is itself Gaussian. So rather than sticking two components together and measuring the noise, or running a computer simulation, I can simply square, sum, and square root.
Each of these theorems reduces the amount of work necessary -- whether by pen and paper or by computer -- by an enormous factor. But if you can describe in similar detail how the computer program would work, I'll acknowledge your greatness.
That's just a few parameters that you avoided having to tune, a few milliseconds of computer time. I'm not great, just questioning basic assumptions that have been handed down from a time before computers were around.
You keep on asserting that the task -- without domain knowledge of Shannon's theorem -- is solvable by computer. Can you describe how such a computer program would work? I'm unconvinced that the computer program would get a reasonable result before you run out of money paying for it.
The Shannon theorem is basically voided by compressive sensing. But all of this (shannon+compressive) was a waste of practical people's time anyway. It's justification for pencil pushers who don't want to do real work.
How about solving differential equations numerically. Such programs are used all over the place in mathematical modeling, with applications ranging from economics to electronics.
And you can't just "write a computer program that tries different parameters". You have to prove that your numerical method solves a certain class of equations first, otherwise your rocket will fly sideways, if at all.
The intelligent way to do that nowadays is to use automatic differentiation followed by optimization "plugins". And that has little to with proving theorems unless you consider all computer programs as proofs and all computer programmers as mathematicians.
I'm not sure what automatic differentiation has to do with solving diff. equations but...
I assume that all those automatic methods you mention are passed down from above in some sort of holy scriptures that we're supposed to blindly believe and use? Or maybe some "pencil pushing" mathematician came up with them first and _proved_ that they actually work?
If you're solving DE's by hand you were probably just in a class taught by members of the Mathematician-Teaching Complex (allusion to Military-Industrial Complex).
I'm curious - you seem to be saying that theorems are useless, and instead, all we ever need to do is write programs to twiddle the numbers and work things out that way.
So what would you need me to do to show that proving theorems is important? What would convince you? What is your criterion for a successful response?
What is the criterion you would give an astrologer? I'm not equating them with astrology, but simply pointing out the countless billions wasted on mathematics that narrows to the rarefied "breed" of pure mathematician earning a salary is as much of a waste as if billions were spent on buggy whip boys. Some things are just meant to fade into museums and history books.
There is no argument that fundamental research (including, but not limited to pure mathematics) is a high-risk investment with a small chance of a large pay-off. But, like most good, high-risk investments the potential pay-offs can be massive and, importantly, not always well-understood in advance.
Would we have GPS without relativity which in turn depends on non-Euclidean geometry, initially considered a curiosity? What about public key encryption, built upon number theory, proudly described as "useless" by the pure mathematician G.W. Hardy? There are so many examples like these.
Be careful of committing the fallacy in assuming that because a clear line of ideas can be traced back from the present day that finding that line was as easy as recounting it.
Out of interest, what is your position of the spending on sport, the arts, and other cultural endeavours?
People are tired of me criticizing maths on here so I'll stop with it.
My opinion on sports, arts etc. is the same. Publicly funded stuff is basically for the elite anyway, it's their way of cleverly siphoning tax-payer money (for there boring outdated interests which they partake in to show the illusion of sophistication). The masses pay to see their interests (and usually heavily taxed for it to boot).
I've had discussion with earnest creationists who point at the fossil record and say "Look - there's no way you can get from A to Z. There's a huge gap." When something is then found that's an intermediate form, they then say: "See! I told you! Now you have two gaps, and it's even worse!!"
So my point is this. I've read what you say, and I can see that you earnestly believe that pure mathematics and the theorems it produces are of no use. I'm asking what it would take to change your mind. What evidence would you need to convince you?
Science is grounded on falsifiable conjectures, and makes progress by testing those conjectures. If you are unwilling or unable to tell us what would be sufficient evidence, then I am unwilling to start chasing ghosts. It seems to me that trying to convince you would be like trying to nail fog to a wall.
Engineering is a critical discipline, and without it, things wouldn't get made. However, engineers rely on techniques that are known to work, and often that knowledge is based on deep theoretical work. Error-correcting codes, via which we get images back from Mars, and which allow reliable communications over cost-effective links are based on theoretical work. Yes, people played about and found the principles, but then they leveraged work done a century earlier to get to the limits. Having done so, they explored the theorems to see what axioms they were based on, and worked to see if those axioms could be circumvented.
It's the interplay between pure theory and pure experimentation that gives us enormous benefits, but it appears that you are willing, even eager, to dismiss fields of which you have no knowledge, purely because you can't see how they can possibly be useful.
It's Blub[0][1], all over again, but here we're not on a linear continuum, we're in a richly interconnected web of dependencies.
Another example. Fourier Transforms were first explored as a purely theoretical construct, showing that functions (with appropriate properties) form a vector space, and that vector space therefore has a basis, and that we should therefore be able to decompose functions into a representation relative to some basis. For decades this was a novelty, and then people started to use it for real. The theorems show the limitations, and then the engineers explore what can be physically achieved within those limits. Wavelets are now often used in contexts where the usual Fourier basis of trigonometric waves prove to be less useful, but the underlying theory is identical, and is still applied. And we know it will work, because of the theorems that were proven decades ago. The vector spaces are, by the way, infinite dimensional, and the work to understand the infinities was done as a part of pure math with no obvious applications.
Another example. We know that some problems are equivalent to others, and we know that the current best algorithms for these problems are exponential. We therefore know, for sure, and not just because of experiments, that some instances of some search spaces will be infeasible without a major break-through. As a specific example of that, multiple times people have claimed efficient algorithms for problems known to be NP complete. In some cases I've been able to prove that their algorithms, while possible useful in general, are definitely not polynomial. I can do that because of the theorems I have to hand.
But you won't care, and I can't make you care. I'm not writing this for you, because you appear not to be willing to change your mind, or consider that a field of which you appear to know very little might just be useful.
No, I'm writing this for people who read your comments and wonder. I'm writing this for people who are willing to entertain the idea that things of which they know little or nothing might be useful in ways they can't yet imagine.
You've given examples of old theorems, and also said they weren't actually useful at first. Later practical work lead to someone seeing use in them (but only the a vague way - limits - which could have been and probably found through experiment as well).
The example you gave of your proof doesn't say all that much, you won an argument (you said those algorithms were practically useful anyway).
I'm sincerely sorry if I come across as hostile. I'd just like to see mathematicians own up and change the corruption from the inside. A lot can come from it, you can change the world (you've taken up a lot it's resources, including some of the smartest people).
You are moving the goalposts again and again, which is why I asked you for the criteria necessary to change your viewpoint. It's clear that you won't, and this will be an endless and fruitless exchange. However, Duty Calls[0].
> You've given examples of old theorems, and also said
> they weren't actually useful at first.
We don't know what of today's work will become useful, or how. I can only give you hindsight evidence, because prediction is hard, especially of the future[1].
> Later practical work lead to someone seeing use in them
> (but only the a vague way - limits - which could have
> been and probably found through experiment as well).
It's limits that are especially hard to establish with experimentation. How do you know that you just haven't yet been clever enough? How do you know that this will always work no matter how far you go? Remember the Ariane 5 explosion? The maiden flight of the Ariane 5 rocket, Flight 501, was lost because engineers reused a unit that had worked flawlessly in every previous flight. The limits were not well understood, and experimentation was expensive. These are guiding principles - sometimes understanding the theory helps more than twiddling bits and seeing what happens. Sometimes twiddling bits is enough. Engineering and Theoretical Research should work together.
In my own work there are at least a dozen cases where knowing a theorem has provoked an exploration, that has then turned out to be useful. The investigation would most likely never have started, the techniques never suspected, without that initial theoretical knowledge. Again, engineering and pure math research in combination, but without the pure math to start with, some things might never be found. We can't know that, of course, but for those who have studied math, the connection is clear. For those who haven't, it's harder to see these things happen. It all seems so obvious in retrospect.
> The example you gave of your proof doesn't say all
> that much, you won an argument (you said those
> algorithms were practically useful anyway).
So, you don't get the idea then. I proved something was impossible given our current state of theoretical knowledge, and that if the claims were correct it was an outstanding breakthrough. He proved nothing, and had a system that worked sometimes, and he was unable to know when, and how, it might fail. I predicted - using pure math research - exactly how his system would fail. I could do something he couldn't, using his system that he created.
> I'm sincerely sorry if I come across as hostile.
Hmm. let's see some of the things you've said:
> ... rarefied "breed" of pure mathematician earning a
> salary is as much of a waste as if billions were spent
> on buggy whip boys.
... and ...
> ... it's the full employment act for pencil pushers.
... and ...
> A lot in common with fundamental religionists.
... and ...
> I never tried getting education in it in the first place.
> I had hunch it's useless and the older I get the more
> I think that was correct.
... and ...
> It's justification for pencil pushers who don't want
> to do real work.
Yes, that does seem hostile. In fact, it seems like you never bothered to study math, and now are trying to convince everyone that something you don't understand cannot possibly be useful.
> I'd just like to see mathematicians own up and change
> the corruption from the inside.
Hmm.
> ... you've taken up a lot it's (sic) resources, including
> some of the smartest people.
So some of the smartest people study math and claim that it is a good thing to be doing. You, on the other hand, claim that it isn't. Your position seems difficult. You claim that something you've not studied, and which is studied and commended by some of the world's smartest people (by your own claim) is, in fact, useless.
Again, I'll never convince you, but I thought your comments should not stand without reply. Perhaps you are in earnest and genuinely believe the things you say, but you are arguing from a position of willful ignorance, which makes it hard to take you seriously.
Which "parameters" are you optimizing? Where did the corresponding model come from?
When is a certain model even applicable? How do you implement the search procedure
efficiently? Is the result actually meaningful? Can you expect future predictions
to make sense or did you overfit to your training data?
To avoid any fluff, let's go with a concrete example: some of the best performing algorithms
for segmentation are based on spectral clustering (http://en.wikipedia.org/wiki/Spectral_clustering).
What does an eigenvector corresponding to the second-smallest eigenvalue of the
normalized graph laplacian have to do with random walks? How do you compute it?
I see that you use VW for machine learning. You should ask the lead developer (John Langford) what he thinks of the efficacy of mathematics for machine learning given most of the algorithms it uses were derived from theoretical considerations (e.g., SGD, LDA, reductions, etc.)
Would you describe him as a "pencil pushing fraud" too?
Yes, he's a pencil pushing fraud for the most part (he might even admit it in private). I don't know how old he is but assuming he's been professional for 10 years, his big contribution is the few hundred/thousand lines of VW (he's probably done other work, but let's assume this is a significant fraction). Where did the rest of his time go? If he just decided to bang VW out a decade ago he'd be at it's current state within a month of starting. VW is only useful because it's fast and it works (and that's not due to any theory). It's theoretical considerations are useful only for essentially drawing that 1 month out to years/decades.
I'll ask him whether he considers himself mostly a pencil pushing fraud when I see him in December. I'll also see what he thinks of the claim that VW's usefulness is "not due to any theory". I'll think you'll be surprised given he's written a blog post on this topic titled "Use of Learning Theory" http://hunch.net/?p=496 at his blog which, by the way, is called "Machine Learning (Theory)".
It's easy to dismiss what you don't understand but you should consider the possibility that it is significantly more difficult develop algorithms like those in VW and "bang out" implementations of them that are fast and correct.
First of all, "learning" is a made up word for parameter search, it's kind of a trick to fool funders to think you're doing cool stuff. Second, his entire blog post is about theory being useful only in a crude way (which basically means not useful) and he's outlined useful rules of thumb (that probaly come from experiment). Is time best spent on gathering data and running experiments to show practical usefulness of different algorithms or on pencil pushing? That isn't made clear.
Firstly, I agree that machine learning is effectively parameter search but the name an artefact of history and we both appear to understand what it means so I don't see how this adds to the argument.
Secondly, no, John didn't say learning theory is "useful only in a crude way", he said, "learning theory is most useful in it’s crudest forms". Big difference. And besides, he says right at the beginning of the post that he believes "learning theory has much more substantial applications [than generating papers]".
To be convincing, theory needs to be precise – if you are not careful about what you are talking about it is easy to believe things that are not true. However, what I think John is saying is that value of theory comes from afterwards abstracting away the precision and understanding the main message (i.e., the "crude form") of a theoretical result. In general, I don't think it is possible to get to a convincing "main message" or "crude form" without someone having grappled with the details.
No matter how many experiments you run, you will only ever show that an algorithm works well or not in a typically small finite number of cases. What theory does is look those cases and ask something like, "It seems that every time X is true of my problem, an algorithm with property Y works really well. I wonder if that is always true?" This type of question gets carefully formalised and then (hopefully) answered. The process of formalisation (i.e., defining things carefully) can yield new ways of thinking about things (e.g., over fitting and bias-variance trade-offs), and having an answer to the general question means that you can be assured that future uses of your algorithm will behave as expected.
You seem to have an unshakeable opinion that mathematics/theory/"pencil-pushing" is inherently a waste of time. That's a real shame. Why do you believe the pursuit of precision, insight, and proof are somehow inferior to running experiments? I find both to be valuable and the interplay between the two extremely rewarding.
'Proving Theorems' is a grand sounding statement, but the reality need not be so glamorous! A Theorem as a thing is an encapsulation of a granule of knowledge or, even less precisely, something learnt which is known to be true[1]. To prove a Theorem is to show that the knowledge it encapsulates is correct. Why is this good?
Most theory I have read[2] on AI, machine learning, and knowledge deals in depth with the concept of symbols and symbol manipulation. Some would claim (correctly!) that computers are in essence simply symbol manipulation machines. The thing is, these symbols are really just ways of encapsulating a piece of information. Often, symbols are described as being 'made-up' of other symbols; symbols are used to describe other symbols. As a simple and obvious example:
- bits (low level 'axiomatic' symbols) are organised into groups to create bytes (still low level, not much meaning here yet)
- bytes are associated with letters (in an isomorphic relation which imparts meaning[3])
- letters are organised into groups to form words (a significant amount of meaning has now appeared)
- words are strung together in sentences
- sentences are woven into paragraphs/chapters/themes/poems/stories/epics
If I was to claim that a poem or story has no meaning simply because I can craft it from simple bits I would be missing something very important about symbol manipulation.
When I take a collection of ideas, snippets of information and learning, and am able to label them with a name I can transcend my current level of understanding and deal with ideas which are invisible on the lower level. For a programmer, this hierarchy of symbols presents itself as a hierarchy of low, medium and high level languages. Writing a python program in assembly would be absurd!(although perhaps appropriate in some cases)
Theorems (and Axioms) are the symbols of mathematics, and proving Theorems is symbolic manipulation in one of its purest forms. In fact, proving Theorems is the main way in which new 'meaningful'[4] symbols are created.
So why is proving Theorems useful? In short, because it is how new knowledge is created! (in this context, mathematical knowledge)
Is mathematical knowledge useful? I think anyone living in this modern world of ours (and I think that phrase applies to most people in most of humanity's history) can see that at least some mathematics is useful. Many have here argued about the different merits of developing mathematical knowledge (ie proving Theorems) before an application is well known, so I won't labour that point. Instead, I think I'll direct the interested reader to G.H. Hardy's 'A Mathematician's Apology'[5] for a great essay on the many merits a joy of mathematics can bring the suitably inclined soul.
If knowledge is a good thing, and proving Theorems increases knowledge, than proving Theorems is a good thing. To say it is better than other ways of spending our efforts is hard, but I think we can definitely agree that it is useful.
[4] An idea again borrowed from Hofstadter. We can string any string of words together that we want, doesn't mean that they mean anything to us. Formal system such as mathematics give us a way to combine symbols to give us new meaningful symbols. My claim that it is the main way they are created is founded on wild speculation and heavily biased historical observation.
You might be interested in the up and coming fields of 3D video games, flight simulators, CG movies, machine learning, supply logistics, autonomous navivations, colonization of Mars, and your personal favorite algorithmic trading, all brought to you by "theorems" of linear algebra.
Great news, I finally wrote a sorting algorithm that worked for 10/10 of my test cases. Now, I can just plug that number into my unproven statisticle test, and found out how likely it is that my algorithm will work for all inputs.
It does not seem to run extra slow for any input, so it probably won't run slowly on any input in actual use. Besides, even if it did I would have no way of proving what types of inputs it failed on, or if another algorithm would do better.
The computer industry wasn't built on Turing completeness. It's an evolution of practical technology starting with the invention of textile machinery (but you can also kind of argue it started with abacuses or even lines drawn in sand).
You are right. Evolution has its mysterious ways. But maybe it's only mysterious to us because our math has not evolved yet to a point where complex systems can also be properly described by theorems.
But then what is the use of the maths. We already have the wealth the comes from the technology. The maths doesn't add much (instead siphons off resources).
You're arguing for eating our seed corn. Yesterday's maths drove today's technology. Today's maths will drive tomorrow's technology, unless you have your way.
Also, the common 1st year texts (Anton, Lang, Hoffman/Kunze and Friedberg/insel/Spence) can be found easily for cheap, used. The old edition of Strang I used to have was good too, but some people react really strongly when you bring it up. There's lots of ways ot sequence LA and the needs of EE's, econometrics/game theory, prob/stats and applied mathies are different from physics/math majors. Look at ToC's and read the Holyoke prof's writeup:
Other things that might be nice to include are dot products (projecting one vector onto another as a measure of co-linearity) and rotation matrices (you could keep it to 3x3).
For the mathematicians here. I am taking Linear Algebra classes as a CS undergrad. Can someone recommend a very good book?
I am looking for the kind of book that will make you fell in love with Linear Algebra. For Calculus, I used Piskunovs Differential and Integral Calculus which was miles away from what my classmates were using. That along with Maple help to double check that my stuff was correct proved a good combo. My current Linear Algebra book is an honest book but it is a boring book, it fails to entertain or to amaze or to give you those moments of insight that puts a grin in your face.
I think I just wish I had better books as an undergrad :-(
OK can you explain kernels and their role in transformations (f.e. the dimension formula)? No you can't this way. And it truly has an INTUITIVE explanation (so much more than the intuitive ess of a spreadsheet!).
Why there are things that, for example, block other things when watching TV (2d TV) and where they are. Why they are all "in the same direction"...
This is the first, simplest, off-hand example I can come up with.
Things are blocked "in parallel" (assuming the camera is far away from the objects) because that projection is a linear map (I repeat, assuming "far away focus") and has a one dimensional kernel: 2d + 1d = 3d, which means (flat TV + orthogonal lines = 3d world), that is the dimension formula.
Every now and again I think my highfalutin college courses were an overpriced waste, and then a conversation like this comes along and I see that the fundamentals of my courses are apparently radical and unheard of most other places.
Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers?
/took linear algebra in the math department, then TAed it.
>Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers?
It is absolutely the case that a great many students will not remember seeing material that they definitely were exposed to. As someone who's TA'd physics classes for a few years, I'd often ask a recitation whether they'd covered some particular subject yet in lecture, and they'd often say (as a class) no.
I'd ask the professor later, and of course they had -- but students rarely go into lecture having done the reading or prep work, and so have a piss-poor retention rate.
I entirely agree. People tend to run away from abstract math, with the impression that it's difficult. But the whole point is that a more abstract view of these things makes them easier to understand, not harder.
The problem is that abstractions are designed to make hard problems harder. Someone who has never seen the type of problem that would make a given abstraction useful would not be able to understand it easily. If you work you're way up, then it becomes obvious.
> Someone who has never seen the type of problem that would make a given abstraction useful would not be able to understand it easily.
Whether or not someone is sufficiently motivated to study something should not affect their ability to understand it. Vector spaces and linear maps are actually much easier to understand, and I think that at some point in mathematics (probably your first analysis course) your motivation has to come from the beauty of the theory itself rather than some real-world application.
My point was not that someone would not be motivated to learn the abstraction, it was that the abstraction would not make sense without a reason to use it. That reason could very well be an entirely theoretical application, but why would the concept of expressing a linear equation as a matrix make any sense to someone who has never dealt with more than 2 linear equations at a time?
Sure, but verroq was talking about vector spaces and linear maps, not matrices. The theory of vector spaces over a field is more general and can be applied to much more than systems of equations.
There is nothing quite like trolling a bunch of nerds with the opinion that the entire study of pure mathematics has been a wasted endeavor for mankind...
Newborns don't come out with all knowledge in their heads already. I'm simply alerting younger people to facts, there's absolutely no reason to get mods involved. Otherwise most of HN should be banned (since a lot of it is not completely fresh news). Why alert the mods, are you a scared pure mathematician?
I might be missing something big, but it seems like linear algebra is then just an overly complicated way of describing what should be a simple toolbox of spreadsheet manipulation functions, kind of like an extra module for python. Why is it given such special emphasis then, it would be like teaching the datetime or sched modules.
The other way around -- Linear Algebra is a (computationally) simple way to describe a (computationally) complicated toolbox of manipulation functions across a huge data set/space.
Learning a few functions that deal with spreadsheets is a lot more simpler and faster (and hence useful to a lot more people) then doing linear algebra courses complete with proofs and 18th century symbols.
What will have more societal impact? Should anyone wishing to use a wordprocessor be required to understand turing machines and computer architecture?
I hope the author is reading your comments, because they make it clear that at least one of his readers is coming away with seriously incorrect ideas about linear algebra!
The thing that you are missing is that linear algebra is about much more than matrix multiplication. It is about the properties of linear transformations in general. While matrix multiplication may have simple implementations in spreadsheets, many other concepts in linear algebra don't, for example calculations of and properties of eigenvectors and the corresponding eigenvalues, or the various matrix decompositions (themselves closely related to eigenvalues and eigenvectors). The matrix decompositions in particular give results that not only are mathematically interesting, but are enormously important computationally.
In terms of societal impact, other than calculus, no field of mathematics has had a greater impact than linear algebra. From quantum mechanics to the google search engine to machine learning, linear algebra is fundamental. Spreadsheets are awesome and a brilliant idea, but they are aimed at solving a different set of problems. Important though they may be, they pale in comparison to linear algebra.
My point isn't that it lacks utility, it's that the proofs are not relevant (maybe they are to the teeniest tiniest number of people but I'm not sure of even that - most research is about speeding up calculations on computers). It's that practically it amounts to a library of computer functions. I can type the date function on the command line, how it works behind the scenes is not relevant.
The details of the proofs are perhaps not relevant to a practitioner. Although I trust you would agree that the existence of the theorems are very useful?
You seem to have the idea that the only useful thing to come out of linear algebra are a set of algorithms, and that as long as you have computer routines implementing those algorithms, the proofs that those algorithms work don't matter. There is a grain of truth in this, but the concepts that you learn in a linear algebra course - particularly around eigenvalues and eigenvectors - are very important in providing you the knowledge to choose those algorithms well. And the process of working hard to understand proofs is a great way (perhaps the best way) to make sure that you really understand those concepts. It's also a great way to make sure that you really understand what a theorem means / why a particular algorithm works.
most research is about speeding up calculations on computers
A a large amount of progress in "speeding up calculations" comes from algorithmic advances. These advances aren't being made by people who view linear algebra as nothing more than a set of library routines. They come from people who deeply understand the concepts and proofs behind the important theorems of linear algebra.
"most research is about speeding up calculations on computers"
So, one of the most convenient things that Linear Algebra does is about making things computationally cheap. Without spending too much time since I can't tell if you're a troll :), this branch of mathematics often times provides the easiest way to optimize data of a certain characteristic. People want to save money/time/labor/etc., and a lot of times linear algebra provides the best framework for doing so, and when you have numbers and data on a large scale it often times becomes infeasible to fiddle around with individual cells in a spreadsheet.
RE: relevancy of proofs -- I think a lot of the most interesting and valuable research in this subject is about figuring out mathematical mappings/equivalence of simple LA function to replace laborious procedures in differential calculus and what not. I personally hated proofs when I went through the education system myself, but the value of doing them is to get an abstract and less-biased understanding of things over learning metaphors that may not necessarily apply precisely to any given situation or problem.
(Disclosure: I'm an operations research and math guy doing data science, so bias here.)
It works, but there's no connection between that process and the intuition of a linear transformation; it's just a rote computation. And checking a long string of matrix multiplications to see if they intuitively make sense (shouldn't everything intuitively make sense?) is especially aggravating when you constantly have to interrupt your intuition to switch to a rote calculation.
I never thought to think of the columns of B as vectors that physically travel through A; to think of a dataflow or pipeline from right to left on the page. Sure, it's not a cure-all, but it'll be a useful mental tool to have.
Oh, and it's also an excellent introduction to the subject, although the Linear Operations section gets a bit muddled... first something's not a linear operation, and then it is, wat? Still, an excellent post.