I think you are ignoring the cumulative nature of mutations. As the old saying goes, a walk of a thousand miles begins with one step.
If this process repeats over and over, as it does in life, then why wouldn’t this type of FI start to accumulate until we hit the 500 bit threshold?