To be clear, when a fraudster uses someone else’s card, the victim of the crime is the bank, not the account holder. Ergo, it’s not actually your money the fraudster is spending, it’s the bank’s money.
Any balance deducted from your account will be put back, and you won’t be charged for an overdraft, or missed bills.
Why are people worried about this? The appropriate checks are in place at the banks, to make sure you don’t lose out.
Alternately, many banks allow you to block specific uses of a card, like contactless, or only allow use of the Apple Pay card whilst keeping the physical card locked.