Physical and sexual abuse are not exclusive (or even nearly so) to family...

But they are overwhelmingly more likely to happen in the family.

Re. sexual abuse, RAINN says 34% of child sexual abuse is by a relative. Unless you're replying to GP, specifically saying "incest is more likely to happen in the family" (which is definitionally true, but vacuous), this appears to be mistaken.

I remain to be convinced re. physical abuse, particularly from other children, particularly given what I know of adult care homes.