I don't think it should be a speed issue, it's clearly a good design issue. Creating multiple tables and a view over the top of that, makes the application more flexible. Systems change all of the time, what happens when someone has three addresses or 5 phone numbers. His 1 table design doesn't allow for this. Your design however allows for different phone/address types.
Also what will happen when you need to add different attributes about your contacts. If he wants one big flat-file you can store that in a text document, thus defeating the whole point of having a RDBMS. At my last job it took 4 years of fighting with the old Mainframe guys about good database design, after they gave it a try, they found it was much easier to maintain the code using multiple tables and relationships.
I would push the issue with your boss to maintain good design standards.