I don't think it should be a speed issue, it's clearly a good design issue.  Creating multiple tables and a view over the top of that, makes the application more flexible.  Systems change all of the time, what happens when someone has three addresses or 5 phone numbers.  His 1 table design doesn't allow for this.  Your design however allows for different phone/address types. 

Also what will happen when you need to add different attributes about your contacts.  If he wants one big flat-file you can store that in a text document, thus defeating the whole point of having a RDBMS.  At my last job it took 4 years of fighting with the old Mainframe guys about good database design, after they gave it a try, they found it was much easier to maintain the code using multiple tables and relationships.

I would push the issue with your boss to maintain good design standards.

Good luck

Tom