Like George said, shrinking as a 1 off is not such a bad thing.

High vlf will adversely affect restores and operations that touch the transaction log.

While instant file initialization will help with the data file, it has no impact on the transaction log. The transaction log still needs to be 0'd out for each growth or during the restore.

If you have a large log file, that process will take longer (compounded by too many vlfs). This is probably why the database backup with the smaller log restored faster.