Taking the phone out my pocket takes the same amount of time as taking the card out my pocket.
With the phone, I have to either hold it over the reader with my thumb over the fingerprint reader, wait for the wallet app to load, them wait for it to read my fingerprint; or double tap the home button to open the wallet app and hold the finger a bit longer to read it, then hold it over the reader. This takes about 1-2 seconds before it is ready to read the card.
With the card, I just hold it over the reader and it works straight away.
Very few places other than the Apple Shop accept Apple Pay for transactions greater than £30, so you have to get your card out for them anyway.