I don't know how much actual data you have, but rebuilding indexes is a fully logged task. It looks to me like the DB simply doesn't have enough room to grow to the task. Depending on how the indexes are structured, a 5 GB database can grow the log file to 11 GB during index rebuilds. Just to be clear... doing a shrink after that basically undoes all the work that rebuilding the indexes did because pages get moved during the shrink.

One of the things you can do is to add the setting to "sort in tempdb" to relieve some of the "pressure". I don't know the exact syntax off the top of my head so I'd have to look it up in BOL. Instead, I'll let you look it up.

Of course, that'll also mean that you'll need a pretty hefty TempDB but, to make you feel better, my production boxes typically start up with a 12GB TempDB.