Simulated netboard - future of linked play in Supermodel?

[Ian]

Found this online
https://docs.google.com/document/d/1fJk ... H0VQ/edit#
From here ..
https://www.aussiearcade.com/topic/8114 ... air-guide/

It seems it's possible to capture the jtag data from a device? If so this could maybe help us with sega rally2?

Mame has this, commented but not implemented

Code: Select all

void model3_state::init_srally2()
{
   init_model3_20();

   uint32_t *rom = (uint32_t*)memregion("user1")->base();
   rom[(0x7c0c4^4)/4] = 0x60000000;
   rom[(0x7c0c8^4)/4] = 0x60000000;
   rom[(0x7c0cc^4)/4] = 0x60000000;
   // Writes command 000023FFFFFFFFFE to JTAG, expects result 0x0040000000 (41 bits)
   // Writes command 000003FFFFFFFFFE
   // Writes command 00003FFFFFFFFFFE 248 times
   // Writes command 000023FFFFFFFFFE, expects result 0x01000000000 (?? bits)
}


[gm_matthew]

It turns out that the random crashes I've been experiencing due to nvoglv64.dll were being logged in Event Viewer the entire time. In total it seems I've had 15 of these crashes, including two back in April before I even started to work properly on getting Sega Rally 2 in linked mode.

Every single one is the same reported error (only the pid and tid values vary):

Unable to recover from a kernel exception. The application must close. Error code: 3 (subcode 7) (pid=11396 tid=19436 supermodel.exe 64bit) Visit http://www.nvidia.com/page/support.html for more information.

I tried searching for "nvidia opengl kernel exception" and according to some people it can occur if the workload from the CPU to the GPU is overwhelming the GPU. Assuming this is what is going on here, it suggests that Supermodel might be making too many OpenGL calls which then causes my graphics card driver to panic and force Supermodel to close. In any case, I can't see how the simulated netboard or any of the changes I've made to get Sega Rally 2 working in linked mode could be inducing a surge of OpenGL calls and causing Supermodel to crash.

P.S. I'd suggest moving discussion about JTAG to The Dark Room

[Ian]

Is it just sega rally that crashes? My best guess is it's being caused by incomplete rendering frames. Other games used to crash the renderer sometimes. And the primary cause was because the writes for a single frame would straddle 2 frames, so the renderer would be trying rendering incomplete data.

All games on the model3 have some kind of boot up sequence where they poll the GPU hardware and perform some kind of sync / timing function. They poll the status register, and it flips at 66% of the frame. Why 66%? I've no idea exactly. But any writes after this time will appear in the next frame. What I did in supermodel to make things work correctly was to move irq2 or what we assume is V-blank to 33% of the frame, and at 99% of the frame the frame buffer swaps, and you see the image. This works for basically all games .. except sega rally.

The reason sega rally 2 needs such a high clock speed now where it didn't used to before is the writes will always seem to straddle 2 frames now. I think the issue is the timing function at bootup has been disabled or broken by the jtag patch. So I think if we can fix the jtag stuff it might also solve this problem.

Before we figured out the 66% thing there was just a variety of hacks that were put together just to get games to work at all, and it was mostly guess work.

If we can solve the jtag thing .. and the game STILL crashes, then i'll take a proper look at the issue.

Mame has written in the comments the values it expects back from jtag, so maybe it's not so hard to fix. But I am not too familiar with the jtag code myself.

[gm_matthew]

For what it’s worth I don’t think CPU clock speed has any significant effect on stability; I’ve had crashes running at 100 MHz and had 25 MHz run without crashing (albeit sloooow). I even tried removing a bit of code that was effectively “wait for vblank” and the result was the game running insanely fast, models and textures flickering like crazy, but the emulator didn’t seem to be any more prone to crashing. This leads me to believe that incomplete frame rendering is probably not the main cause of the crashes, though I’m not going to rule it out just yet.

It’s just occurred to me that when running multiple linked machines, I am effectively increasing the load on the GPU, perhaps making it more prone to crashes if the theory about overloading the GPU is correct. I only got my first crash running a single machine earlier today (though it was linked to itself) and on reflection I think I might have had more frequent crashes with four linked machines. Maybe I could try artificially increasing the frame rate to see what effect that has on stability.

[Ian]

I'll upload a patch to log gpu writes and you can see if they are straddling frames

[gm_matthew]

I set up my own little stress test by creating a build that could go up to 120 fps, starting up four instances of it running Sega Rally 2 in unlinked mode and left them to run for just over an hour. By the time I returned two of them had crashed, one of them just a few minutes in, the other exactly one hour later (though I'm pretty sure this time gap is just a coincidence).

I plan on trying this again with other games or with the legacy 3D renderer.

UPDATE: Completed another test with two instances running side by side, each set to 50 MHz and locked to 60 fps. One instance was using the new JTAG patch that I've been using and the other was using the old patch in the current SVN. Both sides had flickering models and textures (because the CPU speed wasn't high enough) but the old patch ran at full speed while the new patch had considerable slowdown.

BREAKING NEWS: I just had an OpenGL crash while running Daytona USA 2, so now I know it's not specific to Sega Rally 2.

ANOTHER UPDATE: I set up another stress test with Sega Rally 2 but this time using the legacy 3D renderer; after roughly one hour all four were still running.

[gm_matthew]

Given that I’ve now experienced the OpenGL crash in a game other than Sega Rally 2, and no one else seems to be reporting random crashes in Supermodel, it could very well be that my GPU is on its last legs and can’t handle all the OpenGL calls that the new 3D renderer is throwing at it. I do have access to another computer; I’ll try my improvised stress test on it to see if it crashes or not.

In any event, I am now confident that my newer JTAG patch is not responsible for the crashes I have been experiencing so I’m intending to commit it to SVN this evening along with a few other minor changes.

I intend to continue discussion regarding frame timing in Sega Rally 2 in The Dark Room since I have linked play in Sega Rally 2 fully working without addressing the frame timing issues. I’m planning to do a bit more testing this evening to see what I can find out.

[Ian]

Here is a log i captured playing sega rally ..

----------swap buffers----------
flush 13
scroll reg 44
WriteHighCullingRAM 33
Tile generator registers 35
scroll reg 10
Tile generator registers 35
scroll reg 10
Tile generator registers 35
scroll reg 10
WritePolygonRAM 53
WriteHighCullingRAM 53
WritePolygonRAM 58
WriteHighCullingRAM 58
Tile generator 67
WriteHighCullingRAM 67
Tile generator 67
WriteHighCullingRAM 67
WritePolygonRAM 84
WriteHighCullingRAM 84
WritePolygonRAM 86
WriteHighCullingRAM 86
WritePolygonRAM 89
WriteHighCullingRAM 89
WritePolygonRAM 92
WriteHighCullingRAM 92
WritePolygonRAM 94
WriteHighCullingRAM 94
WritePolygonRAM 96
WriteHighCullingRAM 96
WritePolygonRAM 98
WriteHighCullingRAM 98
----------swap buffers---------- <--- should happen before the swap buffers
WritePolygonRAM 0
WriteHighCullingRAM 0
WritePolygonRAM 1
WriteHighCullingRAM 2
flush 16 <-------- end frame happens here ..

Sega rally is the only game this happens with. So there is a good chance the renderer is trying to render incomplete frames / garbage

[gm_matthew]

I found this bit of code in Sega Rally 2:

Code: Select all

0x00078838: 0x8009001C   lwz   r0,0x1C(r9)         ; load word from 0x4c219c
0x0007883C: 0x2C800000   cmpi   cr1,0,r0,0x00
0x00078840: 0x4086FFF8   bf   cr1[eq],0x00078838   ; loop if 0x4c219c is not 0

The DEC exception handler (timed to match ping_pong flip) increments 0x4c219c by 1, and the IRQ2 handler sets it to 0. If the game reaches this point before ping_pong flips, it continues as normal; if not, it waits until the next IRQ2.

I think this bit of code was designed to pause if the frame was not completed in time to prevent garbage frames; with my newer JTAG patch (which enables DEC exceptions) and a lower PPC speed there is considerable slowdown compared to the older JTAG patch. However, Supermodel appears to render the incomplete frames regardless.

It's worth noting that Sega Rally 2 ends each frame by waiting for IRQ2 instead of waiting for ping_pong to flip (which is what most other Model 3 games do).

[Ian]

The status bit vs irq2 has long been a mystery. What should happen is if the frame takes too long the game should drop the frame, ie not write the frame at all. This is what all the other games do.

I'll have to try your patch to see how it changes the logic. Is it in this thread?

It's possible I might need some logic to drop a frame, or really draw the last frame buffers again