If you're rebuilding all your indexes, you're modifying just about the entire database, hence any differential run after that, before the next full backup will be large, because they contain changes since the last full backup.

How long does index rebuild take and how often are your diff backups that you say backups during the rebuild are large?