The issue was on the load balancer.
When the service on server2 was stopped on the load balancer side it still showed up as online for couple of secs (less than 5 secs) so it looks like re-routing the request was bit lagging. After couple of refresh on the browser the error went away and the URL was loading  successfully.