Making the Empirical Case for Human Rights

It’s usual basis is in group loyalty and out group aversion. The facts are adapted to fit that psychology, but are entirely dispensable. There is always a way to rationalize taking care of people like us and holding back those that are different. What is the basis for declaring this just “morally wrong”?