I have noticed this and done timing comparisons between different payment methods and card types.
The short answer is that Oyster is by far the fastest, followed by contactless cards that will accept fully offline transactions, then the slowest (that I’ve found) is contactless cards that require online transactions or server checks. The last one includes tokenised payment methods such as Apple Pay or Android Pay.
Monzo also falls in to the online category, from what I’ve been able to gather, the gate has to at least attempt to connect upwards in the TfL system to get an approval, even if it isn’t going out to MasterCard until a few minutes after you enter the system.