I've got exactly the same problem with SQL Server 2008 32 bits:

'Microsoft.Jet.OLEDB.4.0' was working with .xls, but I needed to try the .xlsx. So I've just installed 'MICROSOFT.ACE.OLEDB.12.0' from the official site with "AccessDatabaseEngine.exe (2007)".

Then the utilization of 'MICROSOFT.ACE.OLEDB.12.0' returns time out every time, and 'Microsoft.Jet.OLEDB.4.0' is broken: OLE DB provider "Microsoft.Jet.OLEDB.4.0" for linked server "(null)" returned message "Unspecified error".

Now I can't find how to uninstall "AccessDatabaseEngine.exe (2007)", the addition of "AccessDatabaseEngine.exe (2010)" doesn't repair it. There is Access 2003 on this server but I can't find AccessDatabaseEngine 2003.

I tried to create linked servers but it's the same.