Connections will be refused if we exceeded max worker threads. Look at the SQL Server error log and identify if there are deadlocked scheduler dumps. If there is follow the steps in http://mssqlwiki.com/2010/06/15/how-to-analyze-deadlocked-schedulers-dumps/ identify on what resource your sessions are waiting (Not excatly session. It is thread or session with threads associated with it).

When you have the issue look at sysprocesses for session with Kpid and identify where they are waiting and check if the active sessions (Sessions with KPID) exceeded max worker threads.