SQL Server will not kill or drop any connection/process unless the service is initiated for a restart or referred from the client process

A sleeping session is waiting for its next command from the application, and is most like a pooled connection; in either case it isn't using any resources.

Have you checked whether your system is having some sort of hardware bottleneck or not ?