Does the report have parameters that get their available values from datasets? If so, those datasets have to be populated when it loads. That could explain why it only affects one report - it has more (or longer queries) datasets than other report.