There’s no reason why it should should make it more difficult to discover new functions, but to understand why you need to separate the three questions of novel function, and more complexity, from the question of fitness.
Natural selection has to do with the fitness effects of mutations (natural selection doesn’t “care” how mutations achieve those effects, whether increases or decreases in functions or complexity). Do they increase survival and reproduction? A mutation doesn’t have to produce a new function, or more genes, to help survival and reproduction. It may simply change the degree of some existing function up or down, and this can help survival and reproduction.
And a mutation resulting in a new function might even be deleterious in rare cases. Suppose an existing transcription factor mutates so now it can bind a new place on the chromosome, but this new binding spot happens to block expression of another important gene. In this case, new functionality would be deleterious.
So when it comes to discovering new functions, it doesn’t matter what the fitness effects of the average mutation is. The specific proportion of these mutations that result in novel functions is largely independent of their fitness effects. So when scientists have discovered that most mutations are pretty much selectively neutral, this didn’t change anything about how likely it is for a mutation to result in a novel function.
However, when it comes to the question of complexity, there are classes of high-probability mutations that can quickly result in increases in complexity. And ironically, this tendency for complexification can actually provide the basis for increased speed of discovery of novel functions. It’s basically the scenario I described above in the figure with all the squares.
Think of it this way: Two classes of mutations are thought to be very frequent: Gene duplication and insertion-type mutations(such as transposons), and deleterious point mutations of relatively small effect.
These two types of mutations, in combination, allow for an increased rate of exploration of sequence space. Here the exploration is driven by inherent mutational tendencies. What types of mutations are most likely to occur (and by types I mean what biochemically happens at the molecular level, not their fitness effects). That is the tendency for repetitive segments to undergo duplication, and for transposons to facilitate their own copy and random insertion, combined with the tendency for “degenerative” mutations to occur in these extra gene copies.
This means the number of genes that are exploring sequence space by accumulating mutations can build up over time, leading to an increased rate of exploration of that space because more and more genes are mutating in parallel. Instead of just one gene waiting for new beneficial mutations to also have novel functions, you get lots of copies of genes that just accumulate lots of mutations of relatively small effect, and so with many genes mutating in parallel you get a much higher rate of sampling sequence space for new functions.
So complexity builds up over time while being mostly selectively neutral, but the complexity increases in turn increases the rate of discovery of novel functions.
New functionality still depends on mutations and genome rearrangements, so whether most mutations that are fixed are beneficial or neutral doesn’t make any difference to the probability of discovering a new function my mutation.
It is a well-known result in population genetics that the rate of fixation of neutral mutations is equal to the rate of mutation. This is pretty well explained in this 12 minute video:
There’s not anything special in some “scenario” that has to happen for this to occur, it simply follows mathematically.
