Generally, no, there's no such thing as too slow unless you fall asleep in mid-move. Aside from taoist-proper explanations, there's some "objective" evidence neuroscientists have found -- slower deliberate movements help establish and strengthen neural connections, you are actually building the "scaffolding" for them in your brain, connected to your body and its systems. In the slow mode, you either reinforce and improve on the structure of neural networks you already have, or even build a new "virtual organ" for yourself comprised of those connections -- that's what well-developed specialized parts of your neural networks are, "brain organs" you create yourself. Fast movement is not as efficient for this until the organ is there and operational. It's true for the new neuromuscular skills of all kinds -- be it learning longhand writing, playing the piano, or speaking a foreign language. You start slowly or you won't create that structure or else it will be haphazard and deficient.
Once it's there, however, you're the king of that castle. If you can do it as impeccably in the fast mode as you do it slowly, you can apply the fast mode to whatever situation warrants it. And surprise! -- you may become faster, or much faster, than someone who didn't create that "organ" with a slow practice! Taiji as a fighting art is -- few people realize it these days when it's been so profaned --first and foremost the fastest of them all. Its primary advantage is the uncanny speed at which it can operate, after all those countless hours/years of being practiced slowly. (It was measured in some studies, with some advanced masters throwing punches while machines registered and calculated the speed, and it was something unbelievable -- I might try to re-find those documentaries.)
So the practice phase that is slow, the slower the merrier, does not become obsolete even when you can do the same thing with impeccability fast, you still want to keep working on it in the slow mode, there's no limit to perfection. What's the purpose of the fast movement then? Well, in martial context, you can't kick anyone's ass in slo-mo. I might elaborate some more later to your impact on the body question in relation to fast movement. (I'm in favor of both, slow is mandatory, fast is a really useful add-on...)