There is mountains of evidence that prepubescent children are greatly harmed by sexual relations, regardless of whether it's with adults or not (though the damage is greatly compounded if it is with an adult), either through direct psychological distress or indirectly through promotion of demonstrably maladaptive and self-destructive behaviors. So why do we only criminalize adults? Because it would be silly to criminalize the children - the basis for illegality is that children lack the mental capacity for selecting sexual partners and engaging in sexual behavior, so if two children do so together, it must follow that neither can be held criminally responsible. Not so in an instance where one party is adult, because adults must be assumed to possess the mental capacity for engaging in sexual behavior.
There is a great deal of cultural and individual variation in when children make the change from prepubescent to sexual maturity, however. What complicates things is that sexual maturity falls on a continuum, and that it isn't possible to identify where on the continuum any individual child falls. From a legal and policy-setting point of view, age of consent laws are therefore set to protect those who reach sexual maturity last: the highest-risk common denominator. They are unquestionably moral and necessary in the form they exist today given the information and risk-assessments available, but they will also improve and get less false positives as our ability to objectively measure sexual maturity improves.