New Disk Image Editor

discuss PC<>Acorn file transfer issues & the use of FDC, XFER, Omniflop/disk etc.
crj
Posts: 832
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: New Disk Image Editor

Postby crj » Wed Apr 18, 2018 4:33 pm

jgharston wrote:Even if INF files don't make sense, either now or then, it's been around for more 25 years and any system that doesn't support them will have the failing that it doesn't support them.

Agreed. As I said:
crj wrote:I mean yes, if it's become a de facto standard it has to be supported for interface purposes

It looks like supporting them is a bit of a headache, unfortunately. Coming to the party late, things I'm unclear on include:
  • How do you transform the base filename into the INF filename?
    • Do you replace any existing extension, or append another?
    • If replace, what happens to files with multiple extensions?
    • If append, what happens if the name becomes too long, e.g. on an 8.3 filesystem?
    • On a VFAT or similar filesystem, do you transform the 8.3 name or the long name?
    • If the base filename is 8.3 but the .INF has a long filename, do you derive the case of characters in the long filename from the base filename or the Acorn filename?
    • How do you handle collisions when either you or some other software has already created a file with that name?
    • What do you do if you have no permission to create the .INF file? (This is a particular problem if you want to provide Acorn attributes for a directory but are not allowed to create files in its parent.)
  • How do you transform an INF filename into the corresponding base filename?
  • If presenting a directory to an Acorn system, what are you expected to do with files that have no corresponding .INF? What about .INFs with no corresponding base file?
  • Are we guaranteed that all characters in the host filename for anything with a .INF will be 7-bit ASCII?
  • Do we pair up files with their .INF case-sensitively or insensitively? Or does this vary according to the host filesystem?
  • How are top-bit-set characters represented in the filename field of a .INF file? (As-is? UTF-8 encoded? UTF-16 encoded? Transcoded? Prohibited?)
  • Am I correct that load/exec addresses should be sign-extended from 24 bits to 32 bits if and only if they are represented by six hex characters in the .INF?
  • What heuristic should be used to determine whether a .INF file is a regular Windows INF or Acorn attributes? (Unfortunately, square brackets are legal in Acorn filenames...)
  • What heuristic should be used to determine whether each successive field in the .INF is usable or a non-"standard" variant?
  • Does anything require, expect, or even use, CRCs in .INF files? If I modify the file, do I ignore the CRC, strip it, or update it? (If a CRC is present, is it guaranteed to be the final entry on the first line of the .INF? Is it guaranteed to be denoted by exactly four hex characters even if leading nibbles happen to be zero?)
  • If you modify the Acorn attributes, do you remove any tail of line 1 you don't understand, or do you preserve it verbatim?
  • What about subsequent lines of a multi-line file?
  • Are line-end characters prohibited, optional or mandatory? If editing the attributes should this be canonicalised or left as-is?
  • Is a DOS EOF character (ASCII 26) prohibited, optional or mandatory? If editing the attributes should this be canonicalised or left as-is?
  • How do you decide when to modify an existing .INF file v. when to replace it?

(For background, one reason I'm so careful about this kind of thing, especially as pertains to Murphy's Law, is that in a past life I worked on software that built a database for a directory hierarchy full of music files. Making sure that you could prepare that database on any of vfat, NTFS, HFS or ext3, under any of Windows, MacOS or Linux, then use it seamlessly under any of the others was... a headache.)

crj
Posts: 832
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: New Disk Image Editor

Postby crj » Wed Apr 18, 2018 4:37 pm

jgharston wrote:Any alternatives must start with "what happens with existing systems". So, going back 25 years, "what happens when I use an Acorn system to copy a file from an Acorn filesystem to a DOS disk?" Which, in terms of filenames is: the filename is as close to identical as possible. So, dragging HELLO from ADFS::HARDDISK4 to DOSFS::0 circa 1990, or using RdAcorn circa 1987, or ReadDFS circa 1985, results in a file called HELLO on the DOS disk and visible from DOS as a file called HELLO and loadable from programs running on DOS by using the filename HELLO.

If the aspiration is to interoperate rather than to archive, the question arises: what transformations to text encodings and/or line endings are performed under what circumstances, and how are they subsequently reversed?

crj
Posts: 832
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: New Disk Image Editor

Postby crj » Wed Apr 18, 2018 4:40 pm

jgharston wrote:Also, remember to translate between BBC filename characters and WinDOS characters:
# <-> ?
. <-> /
$ <-> <
^ <-> >
& <-> +
@ <-> =
% <-> ;
And special case, as it's nonreversable: WinDOS SPC -> BBC _

As a related headache, what happens when we encounter an Acorn file called CON, LPT1 or similar? Is there some accepted standard for what name to give it under Windows, and do we apply that standard even if the target isn't Windows, in case the files are subsequently moved? (Bearing in mind that if we have COM1 and COM1.INF under Linux and they get turned into something like _COM1 and COM1.INF when moved to Windows, we lose.)

crj
Posts: 832
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: New Disk Image Editor

Postby crj » Wed Apr 18, 2018 5:24 pm

jgharston wrote:Also, remember to translate between BBC filename characters and WinDOS characters

Incidentally, it's not stated but I assume it is understood that character 96 in Acorn filenames always maps to ASCII/Unicode character 96 (i.e. backtick) regardless of whether the file is coming from 8-bit or 32-bit Acorn systems?

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Wed Apr 18, 2018 7:24 pm

Pernod wrote:I just loaded up a SSD containing four files with the purpose of simply changing the Boot Option from 0 to 3. When I saved this I expected only a single byte to be changed in the image, but no. It had re-ordered all the files in the image. If no files are being added or removed from the image then I'd prefer it didn't effectively re-create the image from scratch.

I'm afraid that's the way the program works. Sorry.

(When an image is read, it is converted into a list of files (or a tree of files and subdirectories). These are all combined back together on save. It would take a major re-write to keep track of where everything was originally.)

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Wed Apr 18, 2018 7:27 pm

Can I just comment that I am fascinated by this discussion of INF files. I am going to wait to see whether any conclusions emerge before attempting to implement any of it in Beeb Image.

User avatar
geraldholdsworth
Posts: 356
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: New Disk Image Editor

Postby geraldholdsworth » Wed Apr 18, 2018 7:39 pm

yellowpig wrote:(When an image is read, it is converted into a list of files (or a tree of files and subdirectories). These are all combined back together on save. It would take a major re-write to keep track of where everything was originally.)

When I read in a disc, I read the entire image into memory - some might argue that it might be better to open the image file and just read what is needed as and when...sort of like how a real BBC does it (I'm assuming here, of course). I therefore open the image, read in the catalogue, and store the image in memory. When a file is required to be downloaded (extracted? loaded?) from the image, I then extract the data from the image stored in memory.
OK, I haven't quite got around to writing back to an image - although I have started. This writing back would then just mean updating whatever has changed (in the case of uploading/saving a file, this would mean writing the catalogue back, but is held in the same order as read). This also means that just changing the boot option just means changing a single byte as opposed to recreating the entire image.
As far as I am aware, DFS filenames are stored in reverse order that they are stored on the disc (i.e. the file at sector 2 will be last in the catalogue, and the first file on the catalogue will be the last sector(s) used on the disc).
Because I only read the catalogue, the information stored will include the file's location in the image (as this is stored in the DFS catalogue anyway).
Gerald Holdsworth
Repton Resource Page
www.reptonresourcepage.co.uk

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Wed Apr 18, 2018 10:07 pm

geraldholdsworth wrote:When I read in a disc, I read the entire image into memory - some might argue that it might be better to open the image file and just read what is needed as and when...sort of like how a real BBC does it (I'm assuming here, of course). I therefore open the image, read in the catalogue, and store the image in memory. When a file is required to be downloaded (extracted? loaded?) from the image, I then extract the data from the image stored in memory.

That would certainly be one way to do it, like a BBC filing system does it. I just haven't done it that way.

My purpose when I wrote the program was primarily moving files from one format to another. (Host to an image, a DFS disk to a DOS one, etc) When doing this, keeping the original disk minimally changed is not relevant.

David

User avatar
sweh
Posts: 1914
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: New Disk Image Editor

Postby sweh » Wed Apr 18, 2018 10:38 pm

jgharston wrote:
sweh wrote:I wonder if we can create something that's backwards compatible. I wonder how many tools currently read the _second_ line of the INF file? So we could have

Code: Select all

FOO_BAR load exec Locked
V2
NAME:FOO BAR

v2 aware code can look for the second line and if it's V2 then read the 3rd line for NAME: then use that as the filename. Existing code will use the FOO_BAR name instead. Potentially this tagged format would allow for future enhancements as well. ("SOURCE:TAPE"?)

Wouldn't even need to look for V2, just look for the existance of second and further lines.

I was putting "V2" there as a sanity check, but also to potentially allow more versions further on down the line (if we see a V3 section but are only V2 compliant then ignore the V3).
Rgds
Stephen

User avatar
tricky
Posts: 2370
Joined: Tue Jun 21, 2011 8:25 am
Contact:

Re: New Disk Image Editor

Postby tricky » Thu Apr 19, 2018 5:32 am

I'm regretting mentioning inf files.
My original intention was only to note that they existed in the hope that you might be able to import them.

Now though, there is a problem to be solved ;)
How about having a line in the inf that is the beeb file name in hex, two "digits" per original byte (probably easier to parse than uuencode etc).

User avatar
geraldholdsworth
Posts: 356
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: New Disk Image Editor

Postby geraldholdsworth » Thu Apr 19, 2018 8:24 am

yellowpig wrote:My purpose when I wrote the program was primarily moving files from one format to another. (Host to an image, a DFS disk to a DOS one, etc) When doing this, keeping the original disk minimally changed is not relevant.

A programmer writes programs to do something that the programmer wants to do. If they then feels it is worth sharing, it is then shared. I certainly did not write my Disc Image code with the aim of turning it into an Image Reader (and, eventually, writer too). Basically, if someone doesn't like the way one program does something, they'll find other alternatives (and there certainly are a number of alternative disc image manipulation programs, both for DFS and ADFS, around now).
One thing you might want to consider adding in is support for Watford DFS discs. Just a thought.
Gerald Holdsworth
Repton Resource Page
www.reptonresourcepage.co.uk

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Thu Apr 19, 2018 4:16 pm

In an earlier posting onthis thread:
jgharston wrote:Probably just a display formatting issue, but I
copied dfs:RGBEDIT to adfs:RGBEDIT - worked ok.
then copied adfs:RAMCOUNT/S to dfs:RAMCOUN - copy worked ok, but the RGBEDIT entry changed from being displayed DFS style:
* $ RGBEDIT FFFFFB56 0002B070 001AB
to being displayed ADFS style
* RGBEDIT WR FFFFFB56 0002B070 001AB 00

There is definitely a display bug here. I have just come across it myself, but I don't know quite what I did and I haven't been able to reproduce it. I wil correct it as soon as I can pin it down.

David

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Thu Apr 19, 2018 4:23 pm

geraldholdsworth wrote:A programmer writes programs to do something that the programmer wants to do. If they then feels it is worth sharing, it is then shared. I certainly did not write my Disc Image code with the aim of turning it into an Image Reader (and, eventually, writer too). Basically, if someone doesn't like the way one program does something, they'll find other alternatives (and there certainly are a number of alternative disc image manipulation programs, both for DFS and ADFS, around now).
One thing you might want to consider adding in is support for Watford DFS discs. Just a thought.

Thanks for this. It is very much my approach.

Thanks also for the suggestion about the Watford DFS. I suspect this wouldn't be too difficult from where I am, and I am happy to have a look at it when I get time.

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Fri Apr 20, 2018 8:36 am

yellowpig wrote:There is definitely a display bug here. I have just come across it myself, but I don't know quite what I did and I haven't been able to reproduce it. I wil correct it as soon as I can pin it down.

Found the problem. (It happened when you pasted something from the clipboard more than once onto different disks - the two pasted files were not truly independent of one another (if that makes sense))

Also found another problem that would only arise if you tried to import a directory (originally from ADFS or DOS) via the host into DFS.

Corrected both of these and updated the version on the website.

David

User avatar
yellowpig
Posts: 38
Joined: Sat Apr 08, 2006 6:28 pm
Location: Nottingham, UK
Contact:

Re: New Disk Image Editor

Postby yellowpig » Sat May 12, 2018 3:17 pm

I mentioned earlier that I was happy to share the source code of Beeb Image, once I had tidied up a few loose ends and added some comments to the code. I have now done this, so am ready to make it public.

A couple of questions, though, that some of you more experienced folk may be able to advise me on:

  • Where should I put it? Most folk seem to use Github, but I am not sure quite how that works. I presume I need to be registered for it. If so, who does the registering?
  • Do people think it is a good idea to include some sort of "free licence"? If so, which? I can release it under GNU GPL, or is that more than necessary?
David

User avatar
danielj
Posts: 5973
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: New Disk Image Editor

Postby danielj » Sat May 12, 2018 3:40 pm

With github you can set your own account up on and put whatever you like on it. You can also join the stardot group (I think Kieran has to add you?).

The GPL means that you maintain the copyright of all your code, anyone else who derives anything from it has to publish their changes to the source too, and everyone has the right to distribute it, but they must make sure that the copyright of the author is recognised. It covers your butt and makes sure that people can't take the wee-wee with your code -> you've given to the team effort, anyone else using the code and distributing a binary based on it must also contribute to the team effort.

https://tldrlegal.com/license/gnu-gener ... -v3-(gpl-3)

Generally I think it's a good idea to use a license like that as it encourages others to play nicely :)

d.

User avatar
simonm
Posts: 183
Joined: Mon May 09, 2016 2:40 pm
Contact:

Re: New Disk Image Editor

Postby simonm » Sat May 12, 2018 7:23 pm

As Daniel says, gpl is fine for community projects like this. I tend to use MIT license for most of my beeb stuff as I'm really not fussed what happens to the little bits of code I'm putting "out there".

For your tool David, I'd suggest something like this to get you started:

1 - set yourself up with a free GitHub account.
2 - create a new repository on your new account
3 - install the GitHub for Windows desktop app
4 - clone the new repo to your local machine (drag/drop the repo URL from your browser into the GitHub desktop app)
5 - copy your project source files into the new repo folder github app created for you on your local machine
6 - GitHub app will auto dectect these new files have been added to the repo, so now you just need to "commit" (check in) these files to your local repo - there's a button for it
7 - finally, click the sync button in GitHub app, and voila! Your project is now open source on the web!

If you want to make it a stardot community project we can do that later.

Good luck! :)