I dislike both. I can't stand
Olivia, it is one of my most hated "real" names (in a different category to all the incorrect spellings or tryndee names) and I really have never been able to put my finger on why exactly. I think its because its too pretty and princessy and I know that doesn't really make sense but oh well. There is nothing fundamentally wrong with it, it just really doesn't do it for me.
I like the idea of
Olive but my first association is the food and my second is the inferior and superior olive in the brain (I'm a psych student). Its a name I wouldn't mind seeing on someone else's kid but I'd feel kind of goofy calling it across a playground.
Weirdly enough,
Oliver is one of my favourite boys names though and I do really like the nns
Ollie and
Liv and even Livvy...or Via actually for
Olivia.
Rosemary and
Olivia sound better as a sibset to me...less foody.
Rosemary and
Olive sound like they are ingredients from the Italian restaurant's menu :-p