I say, stop worrying. I'd rather be a little startled than a little run over.
I am all too aware of the jumpiness among the passed, because oftentimes I get startled in these situations. It happens when I get into my zone, just cruising along in my happy place when some spandex-encased biker -- or a rare feather-footed runner -- invades my nirvana with a nasally, "Passing!"
The key to avoiding this is in the timing. If I hear someone call out their approach and then pass me the next second, my shit will get freaked out. My natural reaction when I hear "on your left" is to look left before I do anything else. If I look left and see the passer already at my shoulder, that's not a very good warning.
On the other foot, if I have few moments notice -- say five seconds or more -- I am less flinchy. Just by the sound of the voice, I can tell that the person is not in my hip pocket. We all need to be aware of our approaching speed and the time it takes for someone to hear our warning, process it and react accordingly. Hopefully, that means getting the fuck out of the way.
However, you might get a different reaction. Another worry came from Some guy named John: "I find that when I say 'on your left' when I'm about to pass someone on my bike or trusty Asics training flats, they more often than not give me a dirty scowl of a look. I've even had several people tell me to go to hell when I warn them of an impending pass. So, it's a dilemma for me: Rather than doing unto others as I'd have them do unto me, I do unto others as they seem to want for me to do unto them."
I'd place this reaction into the category of "you can't please all the people all the time." Some people are just assholes. Forget them.
I try to say thank you when I receive a good passing notice. I will also call out those fuckwads who don't give me a good warning. It's important that we all do our part to maintain proper etiquette on our shared pathways, trails and roads.
I hereby enact, the Five-Second Rule of passing etiquette.
The Five-Second Rule: Give those slower occupiers of your exercise space five seconds notice with a loud, clear voice to warn them that you are passing. Call out, "on your left" or "passing" or something else equally understood, and make sure to pass on their left side as you