possible starter going bad?

Ok here is my issue.  I have a 94 Suzuki Sidekick and the ignition went bad and did not have the money to replace it so instead I put a push button start which works fine.  The problem is that sometimes when I go to start my car (even after it has just been shut off) it has trouble starting back up.  For instance last night I stopped to pump gas no problems, go to start her back up and the SOB wouldnt turnover you could just hear it cranking.  Now could this possibly be the battery or maybe the switch?  Never had problems with the switch before but it could most likely be possible.  It just doesnt make sense that the car will not start after being shut off for less than 5 minutes.  Any ideas or suggestions on how to test anything besides pulling the damn starter off and taking it to autozone would be greatly appreciated, but if push comes to shove I will have my starter tested if no one can help me try to figure out my problem.

THANKS GUYS!

ok check it outI just went to COSTCO which is where I had bought the battery I had in my car before and exchanged it for a better one went out re-installed it and went to start my car.  I cranked it about 5 times before the darn thing would start.  I am now convinced that this is either a problem with the switch OR possibly the starter is going to give out soon.  Any suggestions gentlemen?

thanks, I will take a look at that!