Technical details

[hectic]

Hi Bart et al.

I figured this was an appropriate place to ask a couple of questions about the development of Supermodel (and probably, extrapolating that, the art of emulation in general). One thing, which an emulation author stated was the finding the technical details was arguably the most difficult aspect of emulation (I paraphrase). I'm looking at System 22 (and the Super variant), but can find little information - datasheets of the processor aside) on the chips themselves, so I guess, my question is when you set out to develop Supermodel and you looked at Model 3, how did you go about finding crucial details of the addressing system / OPCODES of the system particularly in respect of the proprietary chips (e.g Lockheed Martin Real3D/PRO-1000)?

Cheers

Jan

[Bart]

Documentation is nearly always available for CPUs/MCUs/DSPs because these are generally too complex to be fully custom (with rare exceptions). In the case of Model 3, the PowerPC, 68K, and Z80 are all standard and very well-documented parts. From visual inspection of the CPU board, we discovered that Model 3 also uses a Motorola bridge controller (MPC105 or MPC106 in some variants) and the Step 1.x boards use an off-the-shelf SCSI controller, both of which have readily available manuals and data sheets. Graphics chipsets -- especially 3D graphics -- are a different story. They are proprietary pieces of hardware and, because of the highly competitive nature of the market, are nearly always undocumented because publishing detailed manuals risks giving away design secrets. These have to be reverse engineered. The Real3D Pro-1000 was no exception.

In the case of an emulator like Supermodel, you begin with the CPU and then start tracing the code, logging where it reads from and writes to. Through careful examination of access patterns and by studying the machine code, you can begin to figure out how things are supposed to work. It's a difficult process. The bulk of the credit for reverse engineering the Real3D has to be given to Ville Linde and Stefano Teso. It was Ville who originally figured out what I think was the most difficult part: the basic encoding of the polygon vertices and the layout of texels in texture map data.

We did end up getting a hold of a Real3D Pro-1000 API manual thanks to a fellow emulator author. I don't recall how it came into his posession but I think it was leaked by a developer who had worked with Real3D and/or Sega. It was not the register-level documentation that we really would like to have had (and which Sega and Real3D never would have distributed to third parties), nor did it have anything to do with Model 3 specifically (which is a customized derivative of the Pro-1000 image generator chipset described by the manual), but it did provide some useful clues because the API operated on scene graph data that very closely paralleled the structure of data actually consumed by the hardware. We had already figured out most of it ourselves and where the document described features we hadn't (and still have not) figured out, it lacked the details we needed, except in a couple of cases. I was able to figure out how fog and the spotlight work thanks to clues in the API manual (and in the case of the spotlight, by comparing with frame grabs from videos of the real thing recorded by Abelardo). These features are still not working 100% correctly but it's a start. Many aspects of the Pro-1000 still remain a complete mystery to me.

[Arbee]

The same thing applies for (Super) System 22, although we've largely figured that out in MAME and hap has the graphics emulation in very good shape on the latest versions. The 68020 and the DSPs are documented, you have to reverse-engineer everything else. (And for the sound on plain System 22, we had to determine that the Namco "C74" chip is actually a Mitsubishi M37702 16-bit CPU, write an emulation core, and trick it into dumping out its internal ROM program).

[hectic]

Thanks for that both. I remember, RB, a post on your mameworld site ages ago of "Namco are insane - there is no reason other than to annoy emulation authors to make a system this complicated" - or something along those lines - when referring to the C74 and its BIOS.

I stated learning C about 3 or 4 months ago - I wouldn't say I was particularly natural at it - but it's a nice challenge for my curious mind (and since I play older games, I naturally use emulation for thinking of examples). The only thing I'm finding is that as I get deeper into it, I solve one question, but this poses another four. I guess my last (I promise!) question to both of you (although I will use an example of Vivanonno) at this juncture would be if you couldn't actually reverse engineer a sound chip - how would you get it to work? For example, Team Vivanonno produced 22.0.3 before the C74 was properly reverse engineered (at least in MAME - and yes, they are separate projects, but I can't imagine it was reverse engineered fully in 2003, surely?), so how would an emulation author go about HLE like this?

[Bart]

RB can provide a better answer (because a. he's an actual software engineer with more experience than this disgruntled EE student, b. is a sound expert, and c. actually knows about System 22) but I can give you a general answer. I don't know about the specifics of System 22 and its sound hardware but a lot of sound subsystems simply play sampled audio and music. On Model 3, the PowerPC just sends a 3-byte command containing the ID of the sound to play and some other parameters. The sound CPU then figures out the starting address in the sample ROM and instructs the SCSP to play it at the appropriate frequency. Music is a bit more complex. It's certainly possible (although it would not be easy on Model 3 to map all the sound commands to samples) to deduce which sound should be played for a given command, particularly if the audio is mostly just piped out as-is without any additional effects added by a DSP. I could imagine a system where the main CPU even sends explicit addresses of sounds to play in the ROM, along with other set-up information (frequency, duration, left/right balance, etc.) In such a case, it would be really easy to trap these commands and play the sounds directly rather than emulating the underlying sound hardware accurately. Presumably, there would be some effects that the sound CPU could apply that would be difficult to deduce, let alone emulate accurately, which is one big motivation for implementing a proper emulation.

[hectic]

Bart, you're a star. That makes total sense - even if it's a purely hypothetical example. Sorry for the inane questions

[Bart]

Out of curiosity, why did you choose to learn C and how are you going about learning it? What books, online resources, and compiler/IDE are you using? What languages do people tend to want to learn these days and what do they want to be able to do with them? I suspect most learning these days centers about JavaScript, Python, Ruby, PHP, Java, etc. for web-based applications. I'm working on a project that is trying to address the problem of learning how to program and am trying to get an understanding of what the market really wants (both individual users and also institutions and companies, if there is really a need to train people to program quickly). I would love to hear thoughts and comments on this.

[hectic]

By day I'm a researcher for local government in the UK (your version of policy research supporting your elected officials here), so learning C is very far removed from my day job. It's very much a hobby.

Why am I doing it? Well, I'm 31 years old and was raised by my grandparents (this is going somewhere, so bear with me). As you can imagine, this upbringing meant that I never really had much exposure to computing at an early age. For example, my first PC was a P4 with XP on it and at the age of 21 I was thoroughly inept with computers. I stuck with learning about Windows, networking, programs etc until I kind of hit a bit of a learning 'wall'. I got to the point where I learned everything I could do about how PCs worked on a day-to-day basis and needed a fresh challenge.

I play a lot of videogames and ever since I discovered emulation back in about 2002, I have been interested in the 'scene'. Learning C seemed a natural progression from my own self-taught knowledge of computers.

Why C and not something like Perl or Java etc? Well, I suppose I admire the technical ability that emulation authors have and would like to have that ability for myself; or, I suppose at least have the ability to appreciate further the abilities that people like yourself, Arbee and Ville have. I have a very curious nature (which I suppose is where you could draw parallels to my employment) and like to understand how things work. Learning C gives me the ability to understand how something I rely on for endless hours of entertainment - works. I would say that I don't really have much of an end goal. It's a bit like asking someone learning German what their end goal is - it's more of a journey and not a destination for me? If pushed into a corner and with a gun pointed at me, I suppose I would like to write a System 22 emulator at some point simply to prove to myself I have learned something new and have something practical to show for it (and Ridge Racer is digital crack for me), but I'm realistic enough to know that it wouldn't happen overnight and emulation of a complex 32bit proprietary hardware system is a laughable goal for someone who has been learning C for such a short period of time. At the moment, I am quite content in learning about addressing systems, endians etc. It's fun.

What am I using for resources? Well, obviously the internet is one of the resources I am using - only a fool would disregard it as a learning tool. But, I am principally working my way through two books:- Learn C Now and Learn C Through Exercises. I find them really quite accessible. In terms of my IDE I am using CodeBlocks, which I find to be perfect for my needs, though I find the debugging tool to be very vague. Throwing up an error when compiling and giving a nonsensical statement telling me what is wrong isn't particularly helpful.

The one problem I have with learning C - and it's not really related to the language per se, is that in learning solus, it is difficult to find somewhere to bounce ideas or problems around in a non-threatening environment. I live with my partner and while she is supportive of my hobbies, she can't really help with some of the queries I have. I suppose therefore, I lack an environment simply to ask questions. I post here and try and help users regarding specs and stuff, but bothering some of you emulation big boys with - what you would consider to be fairly trivial - questions, I feel a bit, I dunno, cheeky/daft/like an annoyance! Perhaps setting up a separate forum folder for general programming assistance may be worth considering if you weren't averse to proletariat scum like me asking daft questions?! The reason I attach myself to the emulation scene in learning C, as opposed to a programming-centric site, is that emulation is something I understand the concept of, and I can use real-life problems, rather than the strict environment of textbook examples. If I can apply principles (such as the sound chip question I posed earlier) to my learning, I find it easier to understand textbook examples in my books. This is going to sound like insincere toadying, but I'll say it anyway, I genuinely don't think that emulation authors appreciate how influential they are to aspiring coders - and while you get plenty of OMG Y IZ SEGA RALLY 2 NT WURKING FIX IT NAO! comments, which must be a little disheartening to say the least; the vast majority of people who use your emulator appreciate the time; effort and complexity of such a project.

Hope that makes sense?

[Bart]

What am I using for resources? Well, obviously the internet is one of the resources I am using - only a fool would disregard it as a learning tool. But, I am principally working my way through two books:- Learn C Now and Learn C Through Exercises. I find them really quite accessible. In terms of my IDE I am using CodeBlocks, which I find to be perfect for my needs, though I find the debugging tool to be very vague. Throwing up an error when compiling and giving a nonsensical statement telling me what is wrong isn't particularly helpful.

I haven't used that IDE. A debugger would be used to find bugs in code that compiles and executes but does not function as you intended (which implies an error in the design of the program as opposed to a syntax error -- incorrectly written code -- that the compiler would choke on and catch). The debugger should allow you to step through the code, displaying the corresponding line in your source code currently being executed, one line at a time. Have you tried using the debugger in this way?

I think for absolute beginners, some sort of visual execution of the code, stepping through line and by line and indicating which variables are being modified (and how) in each step would be a potentially useful tool. When people first encounter concepts like the stack (ie. how can a function be executed multiple times, even calling itself, and never get the local variables mixed up), I think they sometimes don't 'get it' because it's inherently a low-level concept. These things can be visualized.

I'm asking all this (and I may fork this into a separate thread) because there's been a lot of interest in online learning. It's an area ripe for innovation. There are some sites out there now like codeacademy.com which teach people how to program in web-based languages one step at a time. There's also a really cool site called PlayMyCode. Pretty nifty and they've generated a lot of buzz. I had a very similar idea last summer before I became aware of these existing sites. I still think there's space to compete.

I'm a total newb when it comes to web stuff (and quite frankly, I absolutely hate it and think it's incredibly stupid that people are getting excited about web applications that ran faster as native apps on 20 year old computers -- but that's how it's going to be so I need to make my peace with it).

My friend and I are working on something using Ruby. I'm not a fan of it personally but people seem to dig it. I'm much more comfortable with C and C++ and would rather create an environment focusing on those but I think demand is very low because a) it's not easy to develop a full application in these languages quickly and b) they aren't generally used for web-based projects.

The one problem I have with learning C - and it's not really related to the language per se, is that in learning solus, it is difficult to find somewhere to bounce ideas or problems around in a non-threatening environment. I live with my partner and while she is supportive of my hobbies, she can't really help with some of the queries I have. I suppose therefore, I lack an environment simply to ask questions. I post here and try and help users regarding specs and stuff, but bothering some of you emulation big boys with - what you would consider to be fairly trivial - questions, I feel a bit, I dunno, cheeky/daft/like an annoyance! Perhaps setting up a separate forum folder for general programming assistance may be worth considering if you weren't averse to proletariat scum like me asking daft questions?!

Proletariat scum? Oh, come on. I don't mind answering programming questions here. If we had a lot of traffic on the topic, I would create a special forum for it but for now, The Dark Room is the appropriate place.

The reason I attach myself to the emulation scene in learning C, as opposed to a programming-centric site, is that emulation is something I understand the concept of, and I can use real-life problems, rather than the strict environment of textbook examples. If I can apply principles (such as the sound chip question I posed earlier) to my learning, I find it easier to understand textbook examples in my books.

Of course. Everyone has some end goal in mind and yours happens to be related to emulation. Unfortunately for me and my online learning idea, this is too esoteric of a domain to focus on. I think a lot of people are curious about game development, so that's a potential topic to focus on. Older folks hoping to pick up some new skills are (I imagine) interested in building web sites, perhaps to start their own business or to earn money picking up contract programming work. Addressing the latter topic is something I'd like to do but it's diffcult to build that out piecemeal. A complete -- and complex -- solution would be required almost from the get-go.

Learning how to make games might be a more fun way to learn how to code, and might draw a lot of interest. PlayMyCode is exactly what I had once envisioned -- write games and then share them as an embeddable applet. I can think of some small ways to readily improve on PlayMyCode. I would probably use Ruby directly rather than a custom, derivative language. A Version 2.0 could introduce C/C++, which is more germaine to people who really want to write games and learn how computers actually work.

[motke]

Is the aforementioned API manual the same as this pdf?
http://web.archive.org/web/199806100621 ... guide4.pdf