Copyright law is a quagmire that not many would want to wade through since it can be extremely challenging to pin down even in the best of times, so interpreting the concept of fair use is all the more tough when you think about it. After all, what makes up the qualities of fair use? Theoretically speaking, for anyone to make use of copyrighted content without having to get permission from the creator or rights-holder is not written in stone in the federal copyright law, and the judges in court would have a challenging time before arriving at a decision.
It was ruled that the purpose of the copyright law is to ensure that people are driven to be more creative, as such artistic works end up benefitting society. Hence, the Google Books project would come with a far greater social benefit compared to the infringement aspect.