I've been thinking of ways to argue for FPGAs in the utility computing market. "Cost effective computing" is easier to sell now that multi-core offerings are actually "slower" (in terms of clock speed) than previous chips. Intel and AMD already have us all warmed up to the idea that having more cores is better than faster cores. This makes it easier to market FPGAs as compute elements.
I think FPGAs scale better than CPUs in terms of compute density and power efficiency. I also think theat they force a more scalable programming model by virtue of their architecture. Multi-core chips force the question is "how do we manage concurrent processes across a varying size field of processing elements." The answer is closely akin to an FPGA programming methodology. Not surprisingly, there isn't really a good programming methodology for multi-cores or FPGAs.
Most of the "von Neumann" bottlenecks are associated with the cache structures in a microprocessor. Since FPGA's primitive elements are memories I like to think of them as "smart caches" that can maintain data locality better than RAM and can have operations moved to the data instead of the other way around.
I am meeting with Krste tomorrow to get access to the BEE2 board. I will also put up a CVS on fpgaos.com soon.