So we discovered that a SQL Server restart (not agent) fixes the issue. Then it happens again. I used process explorer to check the DTExec process and there was a bunch of waiting threads, mainly mscorwks.dll!CreateApplicationContext stuck on Wait:UserRequest or

Wait:WrUserRequest or Wait:WRQueue. I found out mscorwks.dll is a .Net DLL which doesn't help much. I removed pieces of the package bit by bit until I found that it's our SSIS framework (called BIPA). When I removed the custom logger everything started working again.