It's the first step in a process that will let us push the envelope on size, indexing speed, accuracy, comprehensiveness and other dimensions. The new infrastructure sits "under the hood" of Google's search engine, which means that most users won't notice a difference in search results.
As a user, I'd like to see less of the cruft appear in my search results. Let's hope that Project Caffeine, after helping Google perform search faster, more accurately, and more comprehensively, lets them produce results that are more useful to us as well.