We do not have to. Reality imposes itself on us.
You cannot be entirely rid of assumptions. Here is one. You are a brain in a stimulus pod. Prove me wrong.
I assume that both our perceptions and thoughts are connected to reality, and further, that you, I, and Darwin for that matter, reason based on nothing more.