Why isn’t the flu shot more effective? The answer, of course, is chickens. (And maybe ferrets)
OK, so the full answer also includes issues already discussed, like the variability of the virus and the reality that a multivalent vaccine may not be equally effective against all targets. But chickens are also part of the answer.
Most doses of the vaccine use viruses grown in chicken eggs. Not all strains grow equally well in these eggs, and serial passage in eggs can result in adaptations in the virus to replicate better in eggs. These adaptations can impact the parts of the virus that are antigenic, meaning that the egg-grown vaccine strains may have subtly different antigens than the virus circulating in the wild. This may mean that antibodies against the vaccine strains may not bind as well to the circulating viruses. The influenza A H3N2 strains are more likely to have antigenic changes when adapting to eggs, and so the vaccine tends to be less effective for H3N2.
As for ferrets, one facet of deciding which strains to include in a given year is looking at antigen cross-reactivity. Do antibodies elicited by infection to strain x bind to virus particles from strain y? If the answer is yes (based on a threshold on a quantitative assessment of binding) and y is circulating, then this supports the use of strain x in the vaccine. The antibodies for these tests often come from ferrets who have never been exposed to influenza viruses until they are deliberately infected with strain x for the test. But obviously ferrets are not humans; it is possible that differences in post-translational modification between ferrets and humans may mean that cross-reactivity could be different with human antibodies. Egg adaptation issues are known; I did not find direct evidence of actual cross-reactivity differences for humans and ferrets.