Emulation Request

want to talk about MESS/model b/beebem/b-em/electrem/elkulator? do it here!
crj
Posts: 830
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Emulation Request

Postby crj » Thu Mar 01, 2018 8:37 pm

If you're starting to build something that elaborate, surely it's better to adapt WAVE or OGG to the task rather than start from scratch?

Indeed, a 40/80/160-channel WAVE file would make a lot of sense.

(Now I'm wondering how well FLAC would compress disc image waveforms. Probably adequately!)

ThomasHarte
Posts: 446
Joined: Sat Dec 23, 2000 5:56 pm
Contact:

Re: Emulation Request

Postby ThomasHarte » Thu Mar 01, 2018 8:49 pm

It only looks elaborate; a simplified emulator can ignore all of the complexity without even parsing it.

But for the record I declined to suggest an elaboration of what amounts to my emulator's internal format (i.e. per-track CSW, which is an audio format) because (i) I wanted the branching to acknowledge that this is an observation, not the true data; (ii) it allows emulators that just want to read a byte, feed a byte to do so; and (iii) I'm not aware of any hardware that actually gives an analogue output level. Nothing I've heard of goes beyond announcing flux transition points.

Point (ii) couples with mostly being able to inspect without dedicated tools and with a small file size, but those considerations were secondary.

It's also generally going to be safe to ignore the per-segment bit rate, or to resample to a fixed rate, so it's not even much extra work for internal MFM bit stream emulators.

ThomasHarte
Posts: 446
Joined: Sat Dec 23, 2000 5:56 pm
Contact:

Re: Emulation Request

Postby ThomasHarte » Thu Mar 01, 2018 9:08 pm

Pulled into a separate post properly to separate it...

Maybe a better alternative is a four-layer per-track format:
Layer 1: a decoded byte stream and an encoding.
Layer 2: bit stream patches. A sequence of PCM parts encoded as "replace bits n to m with these aribrarily many".
Layer 3: speed zones. A specification of the bit rates across the track.
Layer 4: alternates. Indicates sections of the track for which alternate readings were encountered, with a reference to some other track that includes the alternate segment.

All with a track table at the front supplying disk geometry and pointing towards the canonical tracks (i.e. as distinguished from those that are supplied only to provide alternates).

Parse and apply as many of those layers as your emulator supports.

EDIT: with just a few minutes' extra thought, this has already become my favourite idea. It's essentially the inverted pyramid as a file format and has the same effect. If you give up reading it half way through, you'll still probably have understood most of the story. It's a deliberate simplification of the truth, followed by a series of refinements.

Coeus
Posts: 705
Joined: Mon Jul 25, 2016 11:05 am
Contact:

Re: Emulation Request

Postby Coeus » Thu Mar 01, 2018 9:55 pm

ThomasHarte wrote:Parse and apply as many of those layers as your emulator supports.


I am not sure being able to partially implement it actually an advantage for exactly the reason I mentioned in my post about FDI. Any format for which support is inconsistent will get a reputation for being unreliable.

So, if you're still convinced a new format is required I think rather than making layers optional it would be better to have a good reference implementation of a decoder, which obviously means then considering the API between that and the module implementing a specific FDC.

How is track stepping handled? I assume the track index you propose points to the start of the track as defined by the index hole but presumably when the head is stepped from one track to the next on a real drive the settle time timer does not always expire co-incident with the index hole. Do real FDCs do the obvious thing and start looking for the requested sector as soon as the timer expires without waiting for the index hole? Does any software protection rely on that?

Finally, I think it would be good to have a "magic value" at the start of the file to identifiy it as a file of this format and space for a small amount of metadata, for example a title, who archived it, when etc. Not war and peace but a little more than will squeeze into the filename and rather like the comments in a Ogg/Vorbis file.

ThomasHarte
Posts: 446
Joined: Sat Dec 23, 2000 5:56 pm
Contact:

Re: Emulation Request

Postby ThomasHarte » Thu Mar 01, 2018 10:38 pm

Track stepping is handled however the emulator handles it, which hopefully correlates to how the original hardware handles it. It's not something you specify in a file format any more than you'd try to explain what happens to the flags register after a CCF in a ROM image.

And to repeat for the purpose of being explicit: I was following up on the thread of what an ideal image format would look like. Part of that is offering developers an easy route in. I am not necessarily advocating that we create a new format. As per my thoughts on FSD, I'd prefer not to if it can be avoided.

User avatar
billcarr2005
Posts: 1184
Joined: Fri Sep 09, 2005 3:01 pm
Location: UK
Contact:

Re: Emulation Request

Postby billcarr2005 » Fri Mar 02, 2018 11:18 am

TLDR version:

When I created the first FSD, of EXILE, way back in November 2012, I was able to "piggy back" on the SSD code of BeebEM because all that was missing, or rather implemented with automatic values added, was the ability to specify the sector IDs. Having the extra 4 bytes of the sector header per 256 byte sector allowed the copy protected game to run.

Over time, when I imaged more disks and discovered other protection, it was necessary to bolt on other minor modifications, for Data CRC, different sized sectors, etc.

I never wanted a Ferrari to do the local drive to shop for groceries :lol:

ThomasHarte
Posts: 446
Joined: Sat Dec 23, 2000 5:56 pm
Contact:

Re: Emulation Request

Postby ThomasHarte » Fri Mar 02, 2018 3:47 pm

I think we've hit an impasse on this.

FSD is just one more file format, especially when we've already got FDI. But FDI isn't well supported and its cache of software is actually now less accessible than the FSDs on this thread so maybe we don't want to keep that. Of the file formats that inherently exist in the world HFE looks like a good choice but doesn't cover protected disks very well prior to version 3, and version 3 has no public specification.

In a new file format we want: lack of complexity, but that seemingly means both (i) easy enough for emulators to adopt; and yet (ii) not containing any optional content. Despite already existing emulators running the gamut from per-disk-controller sector-centric file support to audio-file-per-track, let the controller be emulated as is necessary to consume that.

So I think proposals are that we: (i) do libfdi to try to compensate for FDI's complexity, at the (severe) cost of excluding jsBeeb and everybody else in the future when we've all lost interest and libfdi has rotted; (ii) hang back for HFE version 3 specifications, cross our fingers and hope for the best; (iii) design our own thing, and simply accept that there is no perfect outcome in doing so.

Especially given the FSD to FDI code already written, I guess it's most prudent to look seriously at (i) first, hope that (ii) usurps that, and circle back to (iii) only if (i) proves to be a bad idea and (ii) is either perpetually non-forthcoming or else unsuitable?

billcarr2005 wrote:... Over time, when I imaged more disks and discovered other protection, it was necessary to bolt on other minor modifications, for Data CRC, different sized sectors, etc.

Anything that causes information to be preserved which was not otherwise being preserved is a net benefit; the reason I prefer to plan up front is that elegance is easier to achieve with foresight and it doesn't leave multiple authors trying to keep up with a persistently moving ball, especially in a low-key hobbyist area where there's always something else that time could be spent on. As a potential consumer of the file format, being able to write my code once and then get on with my life is very valuable versus having frequently to update and change because the file format producer has widened their spec.

EDIT: so, in a wider sense, what I'm talking about is decoupling. Design an interface carefully upfront and you minimise the risk of coupling. A file format is an interface, and what we would ideally avoid is coupling playback (to real disks, to emulators, to whatever) and capture.

User avatar
ctr
Posts: 128
Joined: Wed Jul 16, 2014 2:53 pm
Contact:

Re: Emulation Request

Postby ctr » Fri Mar 02, 2018 5:09 pm

As far as I can see fsd2fdi doesn't handle the complex cases of overlapping sector reads and the like. And it will never handle weak bits so a few games simply won't work with this approach.

So there seem to be two options:

    * Create yet another file format that does handle weak bits.
    * Support fsd. Ideally with a libfsd reference implementation.

User avatar
Pernod
Posts: 1139
Joined: Fri Jun 08, 2012 10:01 pm
Location: Croydon, UK
Contact:

Re: Emulation Request

Postby Pernod » Fri Mar 02, 2018 5:21 pm

ThomasHarte wrote:FSD is just one more file format, especially when we've already got FDI. But FDI isn't well supported and its cache of software is actually now less accessible than the FSDs on this thread so maybe we don't want to keep that. Of the file formats that inherently exist in the world HFE looks like a good choice but doesn't cover protected disks very well prior to version 3, and version 3 has no public specification.

I'm currently trying to implement FDI in MAME, just need to get my head around the Huffman compression and the relevance of the various bitstreams. Can I just use the average stream? I'll also implement the medium-level format and test with FSD2FDI.

I'm sure the existing cache of FDI's will resurface at some point, I have them, just needs someone to make them available somewhere. They aren't lost! I did another search for other machines using FDI, and found none, not even the Amiga scene seem to use it, preferring IPF.

Can we afford to wait for a possible HFE solution, probably not. Maybe we should take another look at IPF?

I think jsBeeb is in an awkward position, it is probably the most accurate emulator in many respects but it's current floppy implementation is high-level. Whilst some users seem to want instantly loading software over accuracy it may not be popular to implement low-level. Didn't someone complain that the floppy sounds were causing games to load slowly. I thought it was all about an authentic experience, rather than faking things for convenience.
- Nigel

BBC Model B, ATPL Sidewise, Acorn Speech, 2xWatford Floppy Drives, AMX Mouse, Viglen case, BeebZIF, etc.

ThomasHarte
Posts: 446
Joined: Sat Dec 23, 2000 5:56 pm
Contact:

Re: Emulation Request

Postby ThomasHarte » Fri Mar 02, 2018 6:13 pm

Pernod wrote:I'm sure the existing cache of FDI's will resurface at some point, I have them, just needs someone to make them available somewhere. They aren't lost! I did another search for other machines using FDI, and found none, not even the Amiga scene seem to use it, preferring IPF.

Can we afford to wait for a possible HFE solution, probably not. Maybe we should take another look at IPF?

Having that other thread running on the HxC forums anyway, I'll lodge a query about timing. That might help to answer the rhetorical question more definitively.

On the purely selfish stuff: my position is like yours in the limited sense that I'm working on a multi-machine emulator, so something that already exists has the added bonus of needing to be implemented only once. I think IPFs are used by the Atari ST community too, which is on my mental hit list, so I dare imagine I'm going to have to tackle it anyway.

Perhaps the lack of public availability of FDIs is actually secretly the answer to all this? If they, and the FSDs here, were made available through a website with a conversion script in place then they could simultaneously be accessible both to downloaders and to jsBeeb as though in IPF or HFE or whatever looks like the best thing. It'd be like writing libfdi except it gets deployed only exactly once, authoritatively.
ctr wrote:As far as I can see fsd2fdi doesn't handle the complex cases of overlapping sector reads and the like. And it will never handle weak bits so a few games simply won't work with this approach.

So there seem to be two options:

    * Create yet another file format that does handle weak bits.
    * Support fsd. Ideally with a libfsd reference implementation.

FSD technically doesn't handle weak bits; I could write a sector with no weak bits and deliberately get the CRC wrong, then test that the data is correct (e.g. via an internal hash) but that a CRC error is indicated. Since the CRC is usually written automatically and correctly, that would make copying very difficult: someone trying to reproduce it using home hardware can't just stop the disk motor or issue an errant track step to write a sector with a bad CRC because all the bytes inside the sector have to be correct and nothing happens instantaneously. That would completely defeat the logic of "if the CRC is wrong, then FSD might not really say so, but obviously weak bits are involved".

FDI however, does. Per the documentation: "A pulse is detected as strong or weak using the "index" stream". Damned if I can figure out why that thing is called an index stream but it appears to be a count of the number of times a pulse was detected and the number of times it wasn't. So it's effectively a received probability. In an obtuse and lengthy form.

Pernod wrote:I'm currently trying to implement FDI in MAME, just need to get my head around the Huffman compression and the relevance of the various bitstreams. Can I just use the average stream? I'll also implement the medium-level format and test with FSD2FDI.

The FDI specification does leave a lot to be desired as a piece of technical documentation. I guess that plus the capture tool being shareware might be what did it for the file format. I think you're probably right to stick to the average stream — I think the point of the maximum and minimum streams is to include a record the range of values received, in case one day somebody goes back and finds that using the average hasn't reproduced the true signal? So they can test other conjectures?

Then as above for my understanding of the index stream, but I have no idea whatsoever why it acquired that name. It sounds like it's going to talk about the actual Shugart index signal, but then the discussion implies it's something that should fire after every pulse if every pulse is strong. So it's probably some internal measure used by Disk2FDI? Maybe my comprehension is failing me.

I think your instinct is correct: implement at least the medium level. Are there even any known images that use the low-level encoding? If not then we're talking about trying to determine the meaning of a slightly confusing text with no test material to verify. Which is hard to support as an endeavour.

User avatar
Pernod
Posts: 1139
Joined: Fri Jun 08, 2012 10:01 pm
Location: Croydon, UK
Contact:

Re: Emulation Request

Postby Pernod » Fri Mar 02, 2018 6:42 pm

ThomasHarte wrote:I think your instinct is correct: implement at least the medium level. Are there even any known images that use the low-level encoding? If not then we're talking about trying to determine the meaning of a slightly confusing text with no test material to verify. Which is hard to support as an endeavour.

All current BBC FDI's are low-level (type 0x80), and the test images at http://www.oldskool.org/disk2fdi/support.html are low-level only too. I have not found any high/medium-level FDI's in the wild.
- Nigel

BBC Model B, ATPL Sidewise, Acorn Speech, 2xWatford Floppy Drives, AMX Mouse, Viglen case, BeebZIF, etc.

User avatar
Arcadian
Posts: 2922
Joined: Fri Nov 24, 2000 12:16 pm
Contact:

Re: Emulation Request

Postby Arcadian » Fri Mar 02, 2018 6:48 pm

Hi, it was me that created all of the Acorn FDI files that were previously available via the Acorn Preservation site.

They are contained on a shared dropbox drive (together with the associated media scans) which several people have access to, and I know that a number of us have multiple offline backups. The drive also contains a quantity of FDI files that I hadn't got round to releasing on AcornPreservation, which includes a bunch of Archimedes games (which I recently passed across to Jon from JASPP at the last RISC OS London show).

One of the reasons I've yet to make the 8-bit FDI files available again is because I wanted to ensure that all of the images were 'good'. I'm pretty sure some of them were duff (for example some of the early 40T Acornsoft releases where the media had decayed). Though there's always the possibility that the FDI emulation in B-Em wasn't 100%.
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk

Image
ABug NORTH (Manchester) (19-21 January 2018)
ABug SOUTH (Hampshire) (1-3 June 2018)

ThomasHarte
Posts: 446
Joined: Sat Dec 23, 2000 5:56 pm
Contact:

Re: Emulation Request

Postby ThomasHarte » Fri Mar 02, 2018 7:45 pm

I have also spotted this thread requesting FDI support over on the HxC forums, which includes this link to sample FDIs, including the following examples:
  • FM: Castle Quest, Elite and 'DFS';
  • MFM: examples for 'ADFS', the Amiga, Archimedes, Atari ST and IBM plus images of Bloodwych (target machine unspecified) and something called Diskspare Device Disk (which I think is for the Amiga based on a hasty search); and
  • GCR: a single Apple and a single Commodore grab (GCR is an umbrella term, the two have different encodings).
So those might be interesting for Pernod as I'm sure MAME must have all of those machines; also for us Acorn folk that provides both DFS and ADFS samples.

User avatar
Pernod
Posts: 1139
Joined: Fri Jun 08, 2012 10:01 pm
Location: Croydon, UK
Contact:

Re: Emulation Request

Postby Pernod » Sat Mar 03, 2018 12:07 am

ThomasHarte wrote:So those might be interesting for Pernod as I'm sure MAME must have all of those machines; also for us Acorn folk that provides both DFS and ADFS samples.

Thanks for those, they are all low-level type 0x8n.
- Nigel

BBC Model B, ATPL Sidewise, Acorn Speech, 2xWatford Floppy Drives, AMX Mouse, Viglen case, BeebZIF, etc.