I only give them to customers who want them rather than just being public for all to see.
If you wrote a cheque, you would be giving out more details because you would show a signature too.
Problems might arise if someone knew other personal info about you. Also, the setting up of direct debits from your account is possible if someone knows the system as some direct debits are signatureless. However, you should get instant refunds from your bank if this happens.
As bank transfers/direct debits become more commonplace, I expect the banks to increase the amount of verification required to set one up. Direct debits are already restricted to companies of a certain standing but this is clearly no longer enough.
It would be inconvenient but ultimately, the loss is borne by the bank.