If you do not hear any "click" whatsoever, then it is likely your ignition switch that is getting worn. Usually, if the starter is bad, you will get a single, rather distinctive "click" when you turn the key to the start position.
The starter is located under the intake manifold and is fairly difficult to access. Not your "typical" starter replacement and takes about 8 hours for someone who is reasonably seasoned at DIY. First, eliminate all of the possibilities before going after the starter itself. Be very astute and careful in your observations and you should be able to nail this one down thus saving yourself a bunch of guesswork.
Ok, when it happens again I'll see if I hear a click. I guess I can only wish it is the ignition switch. lol