Ah... one more thing... I don't believe they use NEWSEQUENTIALID to populate the RecordID column. I believe they use just a normal, random GUID to populate it. In that case and considering the average row size (446 for you IIRC... close to 600 for me), a 94% page density is actually too high and will be subject to more page splits than what you really want. You can do one of two things to fix that... 1) stop rebuilding or reorganizing the index and let the "Natural Fill Factor" shake out or 2) rebuild the index with a 70% FILL FACTOR and then stop maintaining it except for maintaining the Statistics on the table/indexes. Since the NCI's will be in similar shape and the system doesn't do deletes (which would affect the "Natural Fill Factor" of every index, I'd just take this whole table (all the indexes) off your index maintenance and just rebuild the stats.
Just so you know that I practice what I preach, I've only done index maintenance on our phone system once several years ago before I realized what a mistake that was. I've also not done any index maintenance on any of my other production boxes for more than 2 years and performance actually got better in the first 3 months and has not degraded over the 2 years. The only index I've rebuilt was on one "audit" table where they do regular deletes that leave almost every page wanting for fullness (they all got down to 52% and it was a quarter Terabyte table). Because it does have an ever increasing Clustered PK, I rebuilt that at 100% FF and rebuilt the NCI's at 70% and saved about 120 GB. But that's the only reason why I rebuilt it was to recover disk space.
I also rebuilt that table by using the trick of rebuilding the CI on a temporarily created file group in the BULK LOGGED recovery model, then rebuilt all the NCIs, and then did a final rebuild back to the PRIMARY File Group and dropped the temporary file group (and, of course, then returned to the FULL recovery model). It took some extra time but was well worth it because it kept the log file from exploding and it kept me from having an extra 125GB of empty space to go on top of the 125GB of empty space that reducing the size of the index caused.
Thank you Jeff!!!!! You hit it all exactly on the head. After your advice and a little more digging I see that the recommendation by the phone system support team to reindex the IR tables was definitely not good without a lot of caveats. We definitely insert a lot of records into this and other I3 tables and rebuilding with 100% fill factor was hurting my overall performance. I'll be taking your advice and rebuilding with a lower fill factor in increments to watch out for any size/growth issues and monitor the indexes without rebuilding them so frequently. I'll then go the script route on index maintenance at a much longer interval and probably when I do archive/deletes of older data only. I was running crazy routines to keep log file growth checked with log backups after hitting each of these larger tables so frequently.
Thanks for all your detailed advice, definitely saved me a some late nights fighting with this...much appreciated!
Apologies for the late reply. Thank you very much for the detailed feedback!