The one time I forgot to use the pre-filter, I had to clean my filters more often for the first month or so after refilling.
What did it take out for you? I cleaned my filter for the first time 2 weeks ago and it was just a little yellow from body oils etc...no real dirt. I guess I don't need one.
