I'll like to find a solution without loading the driver..
To do that you would need a file that can be opened using a driver that is already installed on the server. The drivers for opening text files already exist - therefore you could build a solution that processes any type of text file. Some valid formats would be comma-separated values (CSV), tab-separate values (TSV), pipe-delimited or fixed width files.
This is exactly the same as asking for a solution to read from an Oracle, MySql or PostgreSql database system, but you don't want to install the drivers for those database products. Can't be done...you need the drivers to be able to make the connection.
I would download and install both the 32-bit (x86) and 64-bit (x64) versions on my workstation. On the server - you only need the x64 version, but if you install the client tools (SSDT) on the server you would need the x86 version. The only reason to do that is if you have a requirement to provide access to the tools for development on that server (e.g. for a vendor who accesses the server through Citrix).
To get the appropriate driver - you need to download and install the Microsoft ACE OLEDB xx driver where xx is the version. You should be able to download 16.0 and use that version.