Yes the age of consent is a cultural relative, but all societies accept that there is an age below which sexual activity is wrong; even the Prophet waited until Ayesha was nine before consummating the marriage.
But the truth is that in most societies middle aged men make the rules and young girls do not get much say in how they feel about it. Recent events in Pitcairn are a good case study.