The Jabba design that was used in Jedi wasn't created til that movie's pre-production, as far as I can tell. So at the very least, even if there was a possibility of making Jabba a stop-motion creature in SW, he would not look like the Jabba we know. The difficulty of integrating a stop-motion character and making it realistic, the way the scene was shot, and the fact that the Jabba character was actually played by an actor rather than a placeholder, who was wearing a fully designed John Mollo costume, the lack of a contemporaneous design for a Jabba creature; these things all lean me towards the idea that compositing in a creature effect was something that might have only occurred to Lucas later on, or at the very least was something he considered pre-SW, dropped for the film's actual production, and later reconsidered. Interesting point; I think several of the SE-and-later changes have subtly reduced the extent to which the OT is "from the adventures of Luke Skywalker" while elevating Vader (the ghost scene changes, the added lines in the throne room). Adding the Jabba scene does also remove a moment of the audience's shared experience with Luke "taking his first steps into a larger world" (I think his lines are meant to play off how the audience would think he Falcon looks cool - but maybe they are 'played straight' with the ship looking nothing like a traditional scifi vehicle).