Then it just could be you a lot of subscriptions during certain time frames or not a lot of threads (based on CPUs) to immediately process the subscriptions.
When the subscription job fires, it inserts a row into the ReportServer database Event table. There is a background task (that is limited in threads based on CPUs) that polls the event table. When it finds an event, it inserts a row in the Notification table and then starts processing. You can probably watch the event table and the notification table and get an idea of where it's slowing down.