Network code

[Ian]

Had a bit of a look at network sync, especially in single threaded mode. Basically it happens right in the end of the frame. In debug mode on my machine it only takes up like 2ms, more if it is stuck waiting in a loop. You can see the chance of them being out of sync, is really high. If the connection is missed, it'll have to wait an entire frame duration to receive the data. Maybe if we time slice the network board board a bit as in the image on the right it might solve these issues? Either that or run the network board in it's own parallel thread?

[gm_matthew]

I’ve come up with a second idea to get the emulated netboards run at full speed: send an “empty” packet to represent IRQ5. Strictly speaking, an “empty” packet would be a single packet reporting a size of 0 which I believe the existing TCP functions should support. (If not, they should be easy to adapt.)

Here’s the master netboard main loop:

wait for IRQ5
send test data
receive test data
send/receive game data

And here’s the slave netboard main loop:

receive test data
send test data
send/receive game data

The slave netboard doesn’t wait for IRQ5, so it will immediately try to receive the next packet of test data; on real hardware it won’t arrive until IRQ5 has happened (because that’s what triggers the master netboard to start the test data loop). My idea is that if the master netboard sends an empty packet at the end of the frame, it will tell the slave netboard that it will not receive the test data this frame so it should wait until the next frame before trying again. The slave netboard can then forward the “empty” packet onwards so that any other slave netboards get the same message. Finally, the master netboard should receive the “empty” packet to complete the loop.

During the netboard self-test, it’s pretty much the same process only without sending and receiving the game data.

Each netboard should always send and receive one “empty” packet per frame. If it reaches the end of the frame without receiving an empty packet, it performs an “empty” send/receive; if it receives an “empty” packet while it was expecting to receive data, it immediately sends its own “empty” packet and does not perform any more send/receive operations for the rest of the frame. This way we don’t have to explicitly define which netboard is the master.

So here’s how the loop would work on the master and slave netboards respectively:

wait for IRQ5
send test data
receive test data
send/receive game data
send “empty” packet
receive “empty” packet

IRQ5 happens
receive test data
send test data
send/receive game data
receive “empty” packet (trying to receive test data)
send “empty” packet

[nuexzz..]

theoretically, the master would have to behave like a slave, when receiving ... assuming slaves have a less aggressive wait in terms of waiting for responses .

[Ian]

I think I finally figured out what the cause of the issues is with the regular network code ..
Another log

Master: Frame 1

send duration 0 size 7 offset 128
rec duration 0
send duration 0 size 7 offset 128
rec duration 0
END FRAME

Slave: Frame 1

rec duration 1676
send duration 0 size 7 offset 128
rec duration 0
send duration 0 size 7 offset 128
rec duration 57
send duration 0 size 7 offset 128
END FRAME

So the issue is here right from frame 1. The send/receive pairs must match. So if frame 1 has two send/receive pairs on the master, the slave must also have twos send and receives, but the slave has 3 instead. This is where the delay comes from. The slave has to wait until the master has done the next frame to finish the send/receive pair. The rest of the frames match, ie same number of sends/receives. But we are out exactly 1 pair caused by the first frame. This continues for the rest of the entire session.

I also wonder about this..

send duration 0 size 15872 offset 256
rec duration 0
send duration 0 size 7 offset 128
rec duration 0
send duration 0 size 15872 offset 256
rec duration 59
send duration 0 size 7 offset 128
rec duration 1
NETBOARD TIME 73

There are 4 sends each frame. First send is 15872 to memory offset 256, second is 7 bytes to memory offset 128. But this is repeated. Maybe we are going 2x too fast each frame?
My guess is there should be 2 sends, instead of 4.

[gm_matthew]

I never actually went into detail about why the emulated netboard has poor performance, so here it is.

In the current implementation, IRQ5 is fired four times per frame (on real hardware it is once per frame). Let's call the period between each IRQ5 a "tick", so that there are four ticks per frame. For most games, after each netboard completes initialization the master waits for IRQ5 before sending the test data, while the slave immediately makes a receive command which blocks emulation until the data is received. This means the slave receives data one tick "earlier" than the master sends it; the netboards are slightly out of sync.

Where this hurts performance is when the slave netboard tries to receive data during the last tick of the frame, when the master netboard has already finished the frame and Supermodel starts executing PPC cycles, 3D rendering, audio mixing, etc. before netboard emulation resumes and the data is actually sent. The slave netboard ends up blocking for almost an entire frame waiting for data, hence why performance is almost halved.

[Ian]

That's a classic problem with emulation. Normally processors etc work in parallel off the same clock or similar clocks. But in emulation things are generally run serially.

Does irq5 trigger the send/receive functions? I see in scud it sends two distinct packets every frame.

[gm_matthew]

That's a classic problem with emulation. Normally processors etc work in parallel off the same clock or similar clocks. But in emulation things are generally run serially.

Does irq5 trigger the send/receive functions? I see in scud it sends two distinct packets every frame.

We already know that the master netboard waits for IRQ5 before starting the test data loop - or at least it's supposed to. In our current implementation, the master netboard doesn't actually wait for IRQ5 because the flag is already set; IRQ5 was already fired twice in the previous frame!

At the moment we only fire IRQ4 after an IRQ5 ack, rather than after sending data which would be the correct behaviour, but this is actually what manages to keep the netboards from desynchronizing altogether. Each netboard waits for IRQ4 twice per loop (once for the test data, once more for the game data), meaning they effectively wait for IRQ5 twice per loop as well. Because IRQ5 is fired twice before the loop ends, the IRQ5 flag is already set when the master netboard checks it.

Here's the main master loop as currently implemented:

wait for IRQ5 flag to be set (no wait, as IRQ5 already happened twice)
clear IRQ5 flag
send/receive test data
wait for IRQ4 (happens after IRQ5)
wait for IRQ6
send/receive game data
wait for IRQ4 (happens after IRQ5)
wait for IRQ6

Here's the main slave loop:

receive test data
wait for IRQ6
send test data
wait for IRQ4 (happens after IRQ5)
send/receive game data
wait for IRQ4 (happens after IRQ5)
wait for IRQ6

I was actually able to get Daytona 2 running at full speed (almost; Daytona 2 single-threaded is quite demanding) by hacking the netboard code to skip a couple of IRQ5 waits, thus ensuring the netboards remained properly synchronized.

Back when I was trying to improve the emulated netboard (before starting work on the simulated netboard), I made it so that all of the IRQs were fired accurately (IRQ4 after sending, IRQ6 after receiving) but it ended up breaking the emulation; the netboards would desynchronize because the slave netboard never waited for IRQ5. The issue was how to get the slave to wait for data if it was not due to arrive until the next IRQ5, and that's what my latest idea is meant to address.

At the moment I'm busy working on the simulated netboard to make it SVN-ready, but at some point afterwards I might try implementing my idea.

[S-O]

An inquiry for Ian and others working on Networking emulation:

Once the base emulation for Model 3 Networking is complete and as bug/fault-free as possible, would you consider looking into the possibility of rollback?

https://ki.infil.net/w02-netcode.html

[Bart]

My concern with one machine running 0.0001% slower or faster isn’t about the video board falling behind with rendering graphics; the vsync signal happens regardless. My concern was that ultimately, vsync must be derived from the pixel clock which is defined as 16 MHz - what if, due to age, temperature variations or something else, one system has a pixel clock speed of 16.000016 MHz? That would cause the 0.0001% drift I’ve been talking about.

Maybe Sega used especially accurate clock crystals. Maybe they have some means of compensating for frequency variations. I am certainly not an electronics expert; my theories as to how the netboard works are mainly down to studying the disassembly of netboard code.

On a modern PC it doesn’t matter so much anyway as neither TCP nor UDP require precise timing like the real netboard does.

Timing could be controlled by a signal passed on the fiber optic cable. Nik mentioned once that if one of the interrupts wasn’t fired correctly, a “no carrier” error was indicated. It is quite likely that there is an even lower level encoding protocol on the fiber link.

[gm_matthew]

Timing could be controlled by a signal passed on the fiber optic cable. Nik mentioned once that if one of the interrupts wasn’t fired correctly, a “no carrier” error was indicated. It is quite likely that there is an even lower level encoding protocol on the fiber link.

It's plausible that there's a register that controls whether IRQ5 can triggered by a "pulse" on the receive line, which would when be automatically re-transmitted; each of the slaves could be set to do this while the master sends the initial pulse. I've seen MAME's m2comm.cpp which suggests that the Model 2 netboard might do something similar to this.

I wonder if looking at a Model 2 netboard program might provide some enlightenment...