my jbl bp600.1

The 4 volt setting on the gain dial would be to the far LEFT, or minimum gain position.  Turning the gain clockwise toward a lower input is telling the amp to adjust its power output curve to receive a smaller amount of voltage input.  The higher the voltage input, the less you have to turn the gain.

The deck may put out 4 volts at certain frequencies, but the bass freqs may be less.  60 Hz may come through at 1 to 2 volts, I couldn't tell you for sure.  You seem to be doing the gain setting correctly, as long as the tone source is recorded at 0db amplitude (pro recording) and the deck is at its greatest level of clean output.  If that is all good, then you are correct in turning the gain.  Bear in mind where you're turning it to, though, because it is true that a clipped signal can be very difficult to detect by ear.  If I were in your shoes and were turning the gain full clockwise like that, I would stop and redo the gain setting with a DMM, measuring voltage output.

And be careful playing a sine wave for very long at high volumes through the speaker(s).  It will quickly burn up a voice coil.  While doing the listening-type gain setting, you might also try using some well-recorded music sources too.  That will allow you longer play time for a more relaxed listening test.  Walk away, too, because low sub freqs can often be heard better at a distance.