Unless I'm misremembering the Phantom story (I saw a version of it when I was 10, so it's possible I've forgotten a lot or that they omitted something), the difference isn't the age gap so much as the fact that when the Phantom meets Christine, she has at least started puberty at that point. Whereas when Valjean adopted Cosette, she was what, six?

It isn't as squicky to act as a mentor to a teenager and then want to jump her once she's an adult. But anyone I had known since they were just out of nappies would be on the "I'd rather claw my own eyes out" list.