Of course don't rebuild needlessly; check the stats and only rebuild those indexes that require it.

Also, with a reasonably well-tuned tempdb, you should strongly consider using the:

SORT_IN_TEMPDB = ON

option. This can help reduce fragmentation of the data file as well.

Even then, you may get a nasty suprise -- after you rebuild the indexes, the db file might be close to the same size it was originally! So be sure to check after rebuilding.