Remember that what you say is true on average, but that we still have genetic ancestors going back as far as you want. Any particular genealogical ancestor is unlikely to be one of the genetic ancestors, but some of them must be. Adam could, by chance, be one of those ancestors. Or he could have had some highly selectable trait that became fixed in the population for that reason. Or God could have caused fixation by a miracle.