(and please bear with me, here. This is probably going to get kind of random)
It's actually the name Hadrian with a couple of funky 'y's in there that really drive me insane. The thing is, when someone sees a 'y', they will immediately associate it with a sound. You associate it with an 'e' sound. That's the way most English speakers say it.
Our language is influenced by other languages. French being one of them. Most people, whether they speak French or not (I don't; I speak German), will associate the French letter 'é' with our 'a' sound. Take the common word "éclair". That's AY-clair. Millions of people know that word, so they see Chloé and say, "Aha! It's French, and I know how to say it!"
I guess your question was if klo-AY was dumb sounding, too. I tend to think of it as a totally separate name and it's pretty in its own right.
As for Chloe itself, I've been known to lapse into the original (Greek?) form and pronounce it KLO. Oops! *looks around guiltily*
I've no idea whether that all made sense to you or not, or if I even explained what I was trying to say.