If paver's are installed properly, and I mean properly (there is a wrong way to install them), you shouldn't have any heaving or racking. Driveways are installed with paver's all the time, take a close look at a proper installation, not heaving, no racking, no rutting with cars, SUV's and large trucks driving on them for years.
I would be concerned about installing in the fashion you described. For one, you will be dragging that stone dust into your tub all of the time, and two, it's just plain ugly.
Here is a like to photo showing my paver installation and then the completed job.
[link]http://www.pbase.com/jlefcourt/backyard_project[/link]
Jonathan