It's funny cuz I found myself in quite a similar situation a couple years ago. I had a pioneer DEH-46DH and 4 pioneer speakers in an 89 voyager w/out infinity system. Then when the car got totalled by the tornado, I moved everything to a 96 caravan (w/out infinity). I was pleasantly surprised to find out that I didn't even have to redo the harness for the radio at all. And those perfect-fit radios sure are easy to put in and out.
ANYWAYS, the harnesses in infinity systems are identical to the harnesses in standard systems. The decks output a high level signal in both cases, it's just that in the infinity systems the factory amps will add a little juice to the signal and split it so that it will go to all the 8 or 10 speakers in the vehicle.
As you know, none of these vans have power antennas so you might be wondering what the power antenna wire is there for. It's to turn on the infinity factory amplifiers, and for non-infinity systems it serves no purpose with factory equipment. However, if you have an infinity system and do not connect that wire, the result is usually no sound at all, and not lower quality sound as you seem to be experiencing. So I don't know if that's necessarily your problem, but the wire should be hooked up anyways. It goes to the blue/white wire in the pioneer harness.
Did you use one of those quick harneses that come with plugs on both ends? Or did you use the more common type of harness where you plug in to the factory harness, and then run the bare wire ends to the bare wire ends of the pioneer harness? Also I should note that some of the infinity systems have a GREEN/ black wire that is kinda separate from the main radio harness, but still back there behind the radio. This is the ground for the factory amplifier and must be connected to a ground for the amps to work properly.
It's hard to say for sure whether or not the problem is that the infinity speakers are too old or their amps are getting old and less efficient, etc. For a factory system they are not bad speakers at all, but they don't even compare to most aftermarket speakers and decks. Right now I'm still using the DEH-46dh in the 96 caravan, and have replaced the 2 dash and 2 rear side panel speakers with aftermarket ones, and added aftermarket speakers to the front doors and the rear tweeter pillars, and it's still all running off of the one pioneer deck with factory speaker wiring (they run the same wiring even if the car doesn't come with the infinity system). It sounds a lot better than any infinity factory system I've ever heard. Your premier deck has the same Mosfet 45 amp chip in it that mine does.
Do you know what the 89 voyager sounded like before you swapped out the infinity head unit? Or do you still have the infinity head unit lying around somewhere? If you could do a quick side by side test to see which one sounds better, that would help narrow down the causes for the difference in sound. Most likely it's just that the infinity system in that 89 voyager is 7 years older...