Sure. As John Mercer stated before your post, viral fitness is defined as…
While all environments are subject to some degree of dynamic, nothing in the animal kingdom approaches disease in terms of pace of change. Within a span of weeks, the same virulent virus can go from a steeply climbing rate of reproduction to falling off a cliff, just due to the near extermination of the infected host population. So fitness always has to be contextually defined. Same thing goes for contact between populations which have adapted to a virus, such as smallpox, with another which is immunologically naïve. That can be devastating to the newly exposed people, but the virus itself may be exactly the same. There, both virulence or fitness are contextual, not inherent traits of the virus.
The mode of transmission also influences the relationship between virulence and fitness. Rabies, aside from recent history making novel medical interventions, is 100% fatal once symptoms present, but unlike the common cold is not given to respiratory transmission. The fitness and virulence of rabies is not positively correlated for transmission. The same can be said for STDs, exhibiting symptoms hardly enhances transmission.
It appears that nature just does not operate by some fundamental principle that the more virulent the virus, the more fit. If the language we use to describe nature fits awkwardly, then we must qualify our language and refine our concepts accordingly. In the case of fitness and contagious disease, that means identifying and stipulating the many associated particulars.