Did the vendor sell you the product and say that it would work with SQL Server? If so, IMHO, the vendor should be working to correct their buggy software, not pushing you to another RDBMS.
As far as SQL Server vs. MySQL, I have no real experience with MySQL, but I believe the saying "You get what you pay for". If you don't have anyone you can go to for help with MySQL and you are confident with SQL Server and that the connection problem is not being caused by SQL Server then why change RDBMS's? It certainly does not seem like you can count on quality support from your vendor.