I confess, this is how I obsess.
To answer the question, I can go either way. On some days, when I want to wear a particular watch, I choose clothing colours which are complimentary to that watch. On other days, I get dressed, put on a watch, and realize that they're not complimentary and choose a different watch. Weirdly, shoe colour is the most influental factor for watch selection.
I also appreciate the fact that a monochrome watch like a classic Speedmaster on steel bracelet can be worn with pretty much anything, which is why it gets worn so much.