To be honest it wouldn't bother me at all. I mean I love it on a boy but I think the whole 'she has stolen a boys name' stuff is bullsh#t. I would definitely only use it on a boy and think it's much more handsome on a boy, but if people want to name their daughters James I really don't care.
I know people always complain about this but to me there's usually a reason why certain names become popular for girls too. James is often used as a nickname for Jamie which is really popular for girls, or was popular 30 years ago. I know several Jamies who get called Jaymz or James just as I know several Hayleys and Amys who go by Hayles and Aymz. Other boys names that are common for girls such as Kyle, Ryan and Reese also sound very soft to me so I can see why they're attractive for girls. Doesn't bother me.
On the other hand I wouldn't mind people naming their sons Bevin or Bronwen at all which sound quite masculine to me. I know some people are going to start the 'you wouldn't name your son Jennifer' discussion, but to me that is just plain silly. The name has to work sound-wise. Jennifer doesn't work on a boy to me but Bronwen and Bevin do. Brutus doesn't work on a girl for me but Ryan and James do.
But yeah, I like it much much much better on a boy.