Slow math can absolutely hurt your performance, but it is usually not the only thing the interviewer is evaluating. What often matters more is whether the slow math disrupted the flow of your thinking, your confidence, or the clarity of your communication.
If you eventually got the right answer but looked structured, calm, and transparent in your approach, one slower calculation by itself might not be fatal. On the other hand, if the math issue caused multiple stumbles, uncertainty, or a loss of momentum, it can affect the overall impression quite a bit.
For your next attempt, focus on both speed and delivery. You want to become quicker, but you also want to narrate the math in a way that makes the interviewer trust your process even before you land on the final number.
Do not reduce the rejection to one moment too quickly. Still, math speed is a sensible thing to improve because it gives you more mental space for the rest of the case. You may also find it helpful to read case interview mental math if you want a more structured way to practice this.