The other time consuming build activity in SSIS is when you're building a deployment utility on every build. Depending on how many packages in your solution this could potentially be causing your issue.

Typically when developing/testing in BIDS i'll keep the CreateDeploymentUtility property set to false, to reduce the response time from starting the debug run and the package actually starting.