I don't think there is a comprehensive book, as much of the internal workings, the costing algorithms etc, as far as I know are not public knowledge.
There's a large chapter in SQL 2008 Internals written by Conor Cunningham (one of the guys working on the optimiser), other than that there's some in Inside SQL Server 2005: Query Tuning and Optimisation, there's stuff scattered through Grant's book "SQL 2008 Query performance tuning distilled"
Also, while it's not specifically SQL server, it may be worth doing a search for academic papers on Query Optimisation. Try scholar.google.com as a starting point. It's really theoretical but if you are interested in this stuff it's worth a look.
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)SQL In The Wild
: Discussions on DB performance with occasional diversions into recoverability
We walk in the dark places no others will enter
We stand on the bridge and no one may pass