I went through all the logs and grabbed some info for you guys. The following links gives us the exact scenario which happened at that time.....

Error 1: There was a memory allocation failure during connection establishment. Reduce nonessential memory load, or increase system memory. The connection has been closed.

Error 2: New queries assigned to process on Node 0 have not been picked up by a worker thread in the last 1680 seconds. Blocking or long-running queries can contribute to this condition, and may degrade client response time. Use the "max worker threads" configuration option to increase number of allowable threads, or optimize current running queries.

http://social.msdn.microsoft.com/Forums/sqlserver/en-US/2917ea9b-d5d4-4a5f-839b-9f363041b685/sql-server-is-not-accepting-connections

http://www.sqlservercentral.com/blogs/aschenbrenner/2011/10/25/troubleshooting-threadpool-waits/

We have a 32 CPU on our production server and max worker threads setting is set to 2048. But according to microsoft recommendation it should be 960.