How about most of your genome? Only 10% is under selection, and most of that is purifying selection that gets rid of new mutations. Of the 40 million mutations separating you from a chimp, only a few thousand, at most, result from selection. In terms of sheer volume, drift is the dominant force in evolution.