We have a custom back end process we use to process batch jobs asynchronously through a queuing table. Many of the jobs that are executed in this process run through a stored procedure that executes a SSIS package using the xp_cmdShell. Originally, these were DTS packages in SQL 2000 and 2005 which executed using the dtsrun executable.
Recently we upgraded to SQL 2008, migrated the DTS packages to SSIS 2008, added a cluster to the mix, upgraded to Win 2008, and are running in a 64 bit environment. The SP still executes the packages via xp_cmdshell but now uses the dtexec.exe (32 bit version). About 90% of the time we have no issues. However, in two instances we have had the SQL Agent job "hang" due to a wait type on the xp_cmdshell. The wait type is PREEMPTIVE_OS_PIPEOPS. Each time we have tried killing the spid, however, this created a rollback that appeared could take hours to complete. The only way to release the pressure and allow users to process again was to fail over the cluster to another node.
The resolution here is not a long term fix obviously. I'm currently trying to determine what is causing this wait type, how to prevent it, etc...I'm leaning toward a concurrency issue and considering making the process run single threaded to try and avoid this. However, since this was never an issue in our SQL 2005 / Windows 2003 environment leaning toward an issue with our new environment.
**Our environment is SQL 2008 but unable to find a forum for the sql agent in the 2008 forums so I posted it here for now.