Well, if you are using joy to key you are translating digital inputs to an analog stick, so there's not going to be a smooth transition.
Unless you are mapping the mouse to the stick, but then you would have to play with the values and sensitivity to reach a point in which you feel comfortable. I guess that the issue is that you are not configuring the supermodel.ini file yourself, because the analog range the emulator has per default is actually better than using joy2key.
Also, keep in mind that you need a fresh copy of supermodel AND the ini file, because supermodel UI fucks the ini file beyond any recognition.
I know that you are using an xbox controller, so you should enter supermodel.ini and look for
[ Global ] ;
There are some options for choosing your graphics engine here and stuff. To use an xbox controller, you have to add this line:
InputMethod = 1
In any case, take into account that using an xbox analog stick (and not a joystick) your thumb will have less space to move, and thus, a lesser range of movement, so the cursor will feel jumpy no matter what, whereas if you use an analog joystick, you will have a longer stick to manipulate, more space, and thus you'll be able to make more subtle motions, but it's like every other thing: a steering wheel allows for smoother movement than a stick in racing rames, for instance.
Also, take into account that the latest xbox 360 controllers quality is very very shitty, with an atrocious dead zone (when compared to the first ones manufactured: I have the one that came with my launch xbox and it works better than the new one I bought last year)