Whilst you are right that you are not liable for fraudulent transactions to your account, the disruption to it because people can apply for products your bank details to sites which do not require a signature. Jeremy Clarkson found this out when he published his bank account details to prove there was nothing to worry about doing it and back tracked.
In the end its your account.