I was literally named after
Jem and the Holograms (Jerrica) and my mum calls me
Jem so I'm obviously used to it as a girls name lol.
I think that if you use Gem as an actual first name you would want to pair it with a longer middle name (or perhaps your last name is long. Some suggestions are:
Gem
AriannaGem
AubreyGem
AveryGem
ElizabethGem
Imogen