HO Alternator Problem

I purchased a 160 amp high output alternator in january, when i first installed it the voltage would decrease big time when it was idling at a stop and sometimes would stall out.  I sent it back, the company said that the alternator works fine but they sent me a smaller pulley for the alternator.  Now i am still having the same problem but it's not as bad.  The voltage will decrease down to about 12-10 volts when i'm idling at a complete stop.  But if i hold the gas pedal and raise the RPMs by about 100 from idling it will go back up to 14 volts.  I put a smaller belt on to compinsate for the smaller pulley and I upgraded the power wire from the alternator to the battery to 4 gauge. Sould i upgrade the ground wire from the battery to the body??? Please help, the company isn't really helping me.