Do i give up modelling because I am the 'ugly' one

First of all I have to say I really enjoy modelling and have become more and more enthusiastic about it.

I've always thought I was odd looking, and was asked to model originally and it all went from there. ..I kind of saw it as an extension of my past doing theatre and acting.

But in the past couple of months I've gained more and more negative comments about my looks from photographers and their associates; I've shrugged it off and thought; well, they don't have to work with me at all/again. However, In the past week alone I've had feedback from photographers with the friends saying, how did you get the 'ginger one' or the 'ugly one' from girls aloud modelling for you

So the question is, what do I do..carry on with something I enjoy, 'fight the cause' so to speak for unconventionally attractive women: or do I throw in the towel now? photographers/models/ everyone on here..do you want to see more of the ugly one?