It's possible that your "problem" is that through having been a hi-fi enthusiast for three decades you are a trained listener. Perhaps, instead of falling into the trap of being hung up on sonic differences, you ignore differences that are purely sonic and focus on musical ones?
As for assessing speakers, my experience is that speakers are the hardest major components to assess rather than the easiest. If you compare two sources in different systems then more often than not you'll prefer the same source in each or have no preference. If, however, you compare two pairs of speakers in different systems then your preference will often vary from one system to the next.
Sometimes you will hear a significant difference that results more from system set-up and/or components than anything else. For example, until recently I was using a Marantz PM4400 amp with (less recently) a Marantz CD5400 CD player. My system set-up was unavoidably bad, and when I upgraded from bass-light Wharfedale Valdus 100SEs to Tannoy Mercury F1 Customs the Tannoys' rhythmic performance was so bad that, in spite of the Tannoys being superior in most respects, I found the Wharfedales more enjoyable to listen to. Had I not been an experienced listener I might have assumed that the Tannoys were intrinsically rhythmically poor. Luckily, I was able to guess correctly that the Tannoys were merely shining too much light on the poor bass performance of the amplifier and CD player, a problem that was being exacerbated by the speakers' poor location. Briefly trying the Tannoys in a better location showed that even with the same amp and speakers they would improve given half a chance. My recent upgrade to a Linn pre/power amp, however, emphasised how much the Marantz amp had been holding the Tannoys back. Even with still poor set-up their rhythmic performance became almost as good (in terms of bass) as the Wharfedales' had been.
As for the "golden ears" theory, I don't think one's hearing is that important. When I was young I could easily hear a 5dB signal throughout the so-called normal 20Hz to 20kHz human frequency range, and I could hear sounds above 30kHz. I think 30kHz is probably a more realistic upper point to describe as "normal". Nowadays, my hearing isn't acute, and my upper frequency is just under 15kHz. Certainly, this means I probably can't hear the tortured high-frequency screeches that some CD players used to emit and I am no longer troubled by CD players' "claustrophobically" steep HF filtering. This sort of thing aside, however, I think I am at least as good at detecting differences as when I was young.
[I think I've spent more time correcting typos after posting this than writing it - sorry if anyone was confused by previous edits].