A practical math topic I think all computer science (and engineering, really) students should take is convex optimization.
Often a problem in engineering or computer science can be phrased as minimizing a cost function over some data and/or variable constraints. Take SVM for example, or sparse signal denoising, or template matching. And far too often, people will apply "advanced state of the art techniques" to solve these problems, e.g. neural networks of some exotic kind, graphical models, etc. And while these are great techniques, they need not be applied to convex problems of a reasonable size.
In the examples I listed, I can almost guarantee that a knowledge of how to phrase the model as a convex optimization problem and throw it into an interior point solver is going to be much more effective than running stochastic gradient descent on some user-defined model with way too many (or too little) parameters.
Because when you say a problem is convex, what you are saying is that it can be solved globally in polynomial time (usually on the order of ~20 least-squares problems of size corresponding to number of variables and constraints). There is a sophisticated convergence theory as well. And if you understand the mathematical theory, you get to use this mature technology to solve exactly a whole class of problems you wouldn't think you could.
It's also worth mentioning that a lot of very hard non-convex problems can be approximated and solved well using these methods, and this is a rapidly growing field of research. Here's an example: http://papers.nips.cc/paper/2979-efficient-sparse-coding-alg...
Not that you stated to the contrary, but the reality is that the theory and algorithms for convex optimization doesn't really differ that much from the theory of general nonlinear optimization. Basically, we get different guarantees, but we're still largely applying the same algorithms (Newton based methods combined with things like projection, active set, or interior point methods for inequalities.) Though, I very strongly agree that knowing convex optimization tricks helps this process immensely. If you're going to formulate a problem, please, dear god, choose the convex one if given a choice.
Mostly, this is a long way to say that if we're going to study optimization, we might as well study continuous optimization, which encompasses convex optimization and isn't that much more difficult.
And, yes, this stuff is incredibly useful with applications in most fields of study. I've made a nice career out of it.
Often a problem in engineering or computer science can be phrased as minimizing a cost function over some data and/or variable constraints. Take SVM for example, or sparse signal denoising, or template matching. And far too often, people will apply "advanced state of the art techniques" to solve these problems, e.g. neural networks of some exotic kind, graphical models, etc. And while these are great techniques, they need not be applied to convex problems of a reasonable size.
In the examples I listed, I can almost guarantee that a knowledge of how to phrase the model as a convex optimization problem and throw it into an interior point solver is going to be much more effective than running stochastic gradient descent on some user-defined model with way too many (or too little) parameters.
Because when you say a problem is convex, what you are saying is that it can be solved globally in polynomial time (usually on the order of ~20 least-squares problems of size corresponding to number of variables and constraints). There is a sophisticated convergence theory as well. And if you understand the mathematical theory, you get to use this mature technology to solve exactly a whole class of problems you wouldn't think you could.
It's also worth mentioning that a lot of very hard non-convex problems can be approximated and solved well using these methods, and this is a rapidly growing field of research. Here's an example: http://papers.nips.cc/paper/2979-efficient-sparse-coding-alg...
Resources:
http://stanford.edu/class/ee364a/lectures/examples.pdf http://stanford.edu/class/ee364a/lectures.html https://www.youtube.com/watch?v=McLq1hEq3UY&list=PL7A3953FD9...