I realize that you are asking about scholars of recent generations—but I can’t help but reflect on how this ubiquitous topic of recent centuries has very old roots.
This was never my field of academic specialization and my memories are foggy but the earliest example I can recall would be Gregory of Nyssa, who studied under Origen, fourth century. He speculated that Adam’s physical body had descended from prior animals but God gave him a “rational soul” which made him an Imago Dei human. (Gregory also claimed that humans had a “plant soul” and an “animal soul”, so that the addition of the rational soul made humans soul-tripartite, so to speak. I don’t remember if he had a term for it but that was the idea and so I long ago tagged that term to my memory.)
Around that same time there was a Roman Emperor who was a well-educated philosopher and prolific writer who claimed that Adam had human contemporaries outside the garden. He was part of the Constantine line of Christian emperors but abandoned his Christianity early on, I think. I could only recall “the Apostate” title so I just now looked him up. It was Julian the Apostate.
I can’t recall if these scholars wrote about the Nephilim per se (Gregory of Nyssa probably did) but I’m fairly sure they assumed there were “people” outside the garden contemporary with Adam. [I put the word “people” in quote because the same ambiguities which apply to the English word today could probably be attached to the Greek and Latin synonyms when scholars of long ago wrote about creatures who may have differed in various ways from ancient and modern notions of “Imago Dei humans.” What word is appropriate for a non-Adamic Homo sapien contemporary, especially when such labels are fraught with potentially incendiary implications concerning racism and worse? Must one have the Image of God and the Gregorian “rational soul” to be considered human?]
I never read much of Giordano Bruno’s work but I wouldn’t be at all surprised if he was yet another scholar who wondered if the Nephilim hinted at people outside the garden.
The Age of Discovery as well as increasing knowledge of the great antiquity of various ancient civilizations led some scholars in those centuries to assume that not all humans started from Adam and that Cain got his wife among those non-Adamic contemporary people groups—and that such ancestors could explain the diversity of the tribes which Columbus, Magellan, et al discovered.
The French Enlightenment brought even more speculation about pre-Adamites and I vaguely recall the Roman Catholic hierarchies struggling to deal with some of the hard-to-refute exegetical writings dealing with Cain’s wife and the people who populated the city he built. Obviously, it is not hard to read the early chapters of Genesis and get the impression that non-Adamic peoples lived outside Eden’s garden. Indeed, I can’t remember a time in my life when that discussion was not fairly common, even if only discussed in “safe” contexts outside of one’s local church and Sunday School class. (My point is that not just the scholars were thinking about these topics.)
I have a vague memory of a text from the “Jewish apocrypha”—which was cited by some rabbinical scholars—where Adam and Eve following their eviction from the garden lived in a cave and occasionally had to deal with other tribes in the area. I thought at first that these stories came from The Life of Adam and Eve—but I did a quick check online and couldn’t find that tie. I would be grateful if someone could help me to identify the specific “outside-the-cave tribe” text I’m recalling.
I can recall occasional speculations in the 1970’s and 1980’s about a postlapsarian Adam encountering non-Imago-Dei neanderthals. Because the theologians and science faculty at evangelical and fundamentalist schools who were most interested in these topics usually had to whisper cautiously such speculations, for fear of losing their jobs, it might be difficult to find many published examples. Even today, I wouldn’t want to identify them by name without their permission.
I also remember very well the various conversations on Amazon book review threads long ago about a “genealogical and theological Adam” in the sense of an ancestor who could conceivably appear in the family tree of all humans but not as the sole progenitor of all humans. Several atheist scientists were involved in those discussions [David Levin of Boston University and paleontologist Christine Janis were among them. I don’t recall the others.] but only the Christian participants were all that interested in what was nothing more than interesting casual speculation. Nevertheless, the scientists did concede that such a “theological Adam” of that sort would not be in conflict with the science, especially since it mostly involved speculation about a non-genetic trait called the Imago Dei being passed down by Adam to all of his descendants without interfering with evolutionary processes and the usual genetics. Nobody used the term “genealogical Adam” but it was a similar idea.
Of course, I would never claim that those casual forum discussions originated the “genealogical Adam and Eve” idea. As with the question in this thread’s OP, speculations about Adamic lines and non-Adamic lines, Cain’s wife, Cain’s townspeople, and the Nephilim as well as how to integrate evolutionary biology and Genesis have been percolating for a very very long time. Obviously, nobody even began in the slightest to tackle the GAE topic with the kind of tremendous scientific depth which @swamidass has undertaken. I’m absolutely elated that a concept which has percolated for so long is about to get its due in his soon-to-be-released book. These are exciting times.