I think there are two main questions we have to ask. First, can the data tell us if the claim is true. Second, is the data reliable.
A quick google gives us this definition for evidence: the available body of facts or information indicating whether a belief or proposition is true or valid. This is the first requirement. The evidence has to be able to test a claim. If a fact or piece of information can’t do this for us, then it isn’t evidence. This also requires us to think about how we can falsify a claim. If a claim can’t be falsified by any conceivable fact or piece of information, then it can’t have any evidence that supports it.
Next, we have to judge the reliability of the evidence itself. Is it independent of the claim? Is there potential bias? Can it be verified? As humans, we often use shorthand and context to convey meaning, and the usage of “evidence” is no different. When people speak of evidence they are often referring to verifiable and independent evidence. For example, a defendant’s attorney can have DNA evidence re-sequenced if they don’t believe the results given by the prosecution. That is verifiable evidence. We could also ask if the police planted the evidence, and that is where we look for sources of possible conflicts of interest or bias. We also tend to look for separate sources for the evidence and the claim. If the snake-oil salesman is the only one who has tested the effects of his product, we tend to give it less weight.
Testimony sits in a really interesting middle ground. Some testimony can be verified, some not. Some testimony comes from people who have a stake in the claim being true, and some doesn’t. Human memory is a wonderful but fallible thing, so how far can we trust it? Add to that the fact that people can lie. I would fully agree that testimony is evidence in the broad sense, but there are many factors that we use to judge the reliability of that evidence.