A quick google of this, it looks like CONTEXT_INFO is not a property that is available at connection time. So you would need to SET that after connecting. This could be done application side - when you call OPEN(), you call a stored procedure (preferred) or direct TSQL to SET the CONTEXT_INFO value. Alternately, you could modify the stored procedures to SET the CONTEXT_INFO as the first step of the stored procedure (if this makes sense to your process).
Maybe not the most graceful approach, but from what I was reading, it appears to be the only approach.
It is also being phased out in favor of SESSION_CONTEXT() from what I was reading.
http://www.databaseteam.org/1-ms-sql-server/617a805e0aa24198.htm <-- old link for old SQL version but they did some testing and were unable to set it with the connection string
https://www.mssqltips.com/sqlservertip/4094/phase-out-contextinfo-in-sql-server-2016-with-sessioncontext/ <-- talks about using SESSION_CONTEXT() instead of CONTEXT_INFO() as it offers a lot of benefits in SQL Server 2016
The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!