Running the pumps to power the jets helps heat the water. If your spa lost that much heat without the pumps running, then your spa water was loosing heat faster than the heater could replace it.
If it was windy out, that would further accellerate heat loss. There is a lot of temperature difference between 104 and 30, and the greater the difference the faster the heat loss. Be glad it wasn't below zero!