Should i be concerned of my bfs past?

Well he has an ex that wants him back and kept asking him when will they get back together cause she still liked him even though she cheated so she would text him and go up to him whenever i wasn't around. Then he is my first who i lost my virginity to and i guess it bugs me that he did it with two other people so i just feel like another girl to him even though he says I'm not and that he never has fallen in love until now with me. I mean i know its his past but should i worry about it?