Do you need the exact row counts or can a very close approximate do, if so I would join back to sys.partitions where the index id is 0 or 1 and pull the row count out via that metadata view instead.  The documentation states the values are "approximate" but I have yet to see a difference when querying select count(*) vs the view