What is the client in Azure? What service? Are they in the same region? Are they in the same zone? Are they in a proximity group?
You will need to check that the vNet or the Service has access to the Microsoft.SQL.Database (iirc) network service in Azure otherwise it will be routing out via the internet to come back in, instead of using the Azure backplane.
Azure SQL DB will not see any network latency on the connection, you will need to use something like Wireshark, to capture a packet trace between the client and the SQL DB endpoint