KLO-AY is definitely not stupid, because it's the normal French pronounciation of Chloé, which is the French version of
Chloe, and not some ridiculous ruining of the name, as you seem to imagine. Accents are there for a reason, not just to look pretty.
If there's any stupidity involved, it's in putting an accent that alters the pronounciation and then expect the name to be pronounced as if the accent wasn't there.
If it bothers you so much, I suggest you sign your name without the accent and possibly see if legally you can get rid of it from your official name.