These are really good questions to be asking, and to help with that let's consider 3 attributes of compute complexity: time, space, and memory
The traditional way of computing on a CPU is in essence a list of instructions to be computed. These instructions all go to the same place (the CPU core) to be computed. Since the space is constant, the instructions are computed sequentially in time. Most programmers aren't concerned with redesigning a CPU, so we typically only think about computing in time (and memory of course)
On an FPGA (and custom silicon) the speedup comes from being able to compute in both time and space. Instead of your instructions existing in memory, and computed in time, they can be represented in separate logic elements (in space) and they can each do separate things in time. So in a way, you're trading space for time. This is how the speed gains are achieved.
Where this all breaks down is the optimization and scheduling. A sequential task is relatively easy to optimize since you're optimizing in time (and memory to an extent.) Scheduling is easy too, since, they can be prioritized and queued up. However, when you're computing in space, you have to optimize in 2 spatial dimensions and in time. When you have multiple tasks that that need to be completed, you then need to place them together and not have them overlap.
Think trying to fit a ton of different shaped tiles on a table, where you need to be constantly adding and removing tiles in a way that doesn't disrupt the placement of other tiles (at least not too often.) It's kind of a pain, but for some more constrained problem sets, it might make sense.
These aren't impossible problems, and for some tasks, the time or power usage savings is worth the additional complexity. But sequential optimization is way easier, and good enough for most tasks. However, if our desire for faster computing outpaces our ability to make faster CPUs, you may see more FPGAs doing this sort of thing. We already have FPGAs that are capable of partial reconfiguration, and some pretty good software tools to go along with it.
The traditional way of computing on a CPU is in essence a list of instructions to be computed. These instructions all go to the same place (the CPU core) to be computed. Since the space is constant, the instructions are computed sequentially in time. Most programmers aren't concerned with redesigning a CPU, so we typically only think about computing in time (and memory of course)
On an FPGA (and custom silicon) the speedup comes from being able to compute in both time and space. Instead of your instructions existing in memory, and computed in time, they can be represented in separate logic elements (in space) and they can each do separate things in time. So in a way, you're trading space for time. This is how the speed gains are achieved.
Where this all breaks down is the optimization and scheduling. A sequential task is relatively easy to optimize since you're optimizing in time (and memory to an extent.) Scheduling is easy too, since, they can be prioritized and queued up. However, when you're computing in space, you have to optimize in 2 spatial dimensions and in time. When you have multiple tasks that that need to be completed, you then need to place them together and not have them overlap.
Think trying to fit a ton of different shaped tiles on a table, where you need to be constantly adding and removing tiles in a way that doesn't disrupt the placement of other tiles (at least not too often.) It's kind of a pain, but for some more constrained problem sets, it might make sense.
These aren't impossible problems, and for some tasks, the time or power usage savings is worth the additional complexity. But sequential optimization is way easier, and good enough for most tasks. However, if our desire for faster computing outpaces our ability to make faster CPUs, you may see more FPGAs doing this sort of thing. We already have FPGAs that are capable of partial reconfiguration, and some pretty good software tools to go along with it.
TL;DR: Geometry is hard.