You shouldn't feel upset at all. If anything, you should feel proud! I respect you for waiting for the person you love a lot more than the people who just throw it away because they think it's the cool thing to do. I don't think the age you lose it at is something to be ashamed of if you're older than average. Everyone should hold out for who they feel is the right person. I'm 20 now and only lost my v-card a few months ago to my girlfriend who I have been going out with for over two years. I couldn't care less that I was 20 when losing it even though some of my friends lost it at like 16. It doesn't make me a loser and you definitely should do it to be in the "cool crowd", it'd make you a fool.
Hold out for that person you love and you will be much happier, you will have a more pleasant experience with someone who loves and cares about you.