Latest version of BeebEm

[hoglet]

Could it have something to do with the hardware configuration of the machine that wrote out each individual tape? More frequent IRQs on those machines or something?

Certainly if the ACIA Tx Interrupt is slow to be serviced, then the ACIA master reset write would be delayed.

i.e. there clearly the potential for a race condition between the interrupt handler and the first of the two extra bytes.

This might allow part of the first byte of extra data to "leak out".

Can you say anything about the range of values you see when this extra byte is present. Are the higher numbered bits typically '1'?

The interrupt might be slow to be serviced if another interrupt is pending. But think is unlikely to affect all blocks in the same program equally. It would be pretty random.

I wonder if there is a way the presence of certain paged ROMs might delay the interrupt processing? That might cause the behaviour to be consistent on a particular machine.

Also, it might be different on a Master.

Dave

[Diminished]

I should be making dinner, but apparently this is more important.

So I did indeed manage to find one of the candidate titles within the vanekp WAV tape archive -- Caveman Adventure.

I grabbed "CavemanAdventure(ProgramPower).wav" from this WAV archive, and ran it through Quadbike. By default, Quadbike just produces a CSW output file, but the --inspect-dir debugging option tells it to dump useful intermediate data into various WAV files in a specified directory -- essentially, it allows you to "see what Quadbike is thinking".

Here is the very end of block 0 of the first file on the tape, according to Quadbike.

The top trace here is the 1-bit (2400 Hz) signal power, according to the Goertzel transform employed by Quadbike to identify bits. Below that is a sync stream derived from it, which shows where each pair of half-bits is sampled from the power. The bottom trace is the resulting decoded bitstream, which I have annotated with the bit values. Remember that bytes are sent LSB first, so the bits in each byte are read right to left, resulting in a sequence of DD 7B 7B as the final three bytes in the block.

Similarly, here's block 1:

So we would expect sequences of DD 7B 7B, and 80 31 31, for the final three bytes of the first two blocks on the tape. It certainly looks like there is a single duplicated CRC byte involved here, but what does the hacked Cornfield say about the UEF file from Stairway -- "CavemanAdventure_B.uef"? Does it agree?

Code: Select all

  stardot: blk #1: 1 garbage bytes; first matches CRC MSB (7b :: 7bdd)
  0x0153  4e dd 7b 7b
  stardot: blk #2: 1 garbage bytes; first matches CRC MSB (31 :: 3180)
  0x027d  69 80 31 31

You bet it does.

[Diminished]

Can you say anything about the range of values you see when this extra byte is present. Are the higher numbered bits typically '1'?

Good question. Not really. There seems to be a fairly even distribution for the most part, and all possible byte values seem to appear at least once. However, &FB is overwhelmingly the most common repeated CRC byte (62 occurrences, versus just 34 for its nearest rival, &D1). If I had to guess, I'd suggest that this is just because certain CRCs are more common than others, based on certain code (and therefore blocks) being shared between multiple tapes?

For completeness, here's a list: the frequency first, followed by the byte value it applies to, in case anyone wants to plot a graph or anything:

Code: Select all

  32 00
  18 01
  19 02
  16 03
  27 04
  20 05
  17 06
  27 07
  18 08
  21 09
  25 0a
  15 0b
  20 0c
  24 0d
  14 0e
  14 0f
  24 10
  25 11
  27 12
  16 13
  22 14
  18 15
  32 16
  21 17
  12 18
  20 19
  16 1a
  21 1b
  24 1c
  26 1d
  17 1e
  21 1f
  23 20
  19 21
  18 22
  32 23
  17 24
  10 25
  25 26
  22 27
  15 28
  20 29
  21 2a
  15 2b
  30 2c
  20 2d
  17 2e
  15 2f
  18 30
  19 31
  22 32
  22 33
  27 34
  21 35
  25 36
  23 37
  17 38
  18 39
  16 3a
  20 3b
  12 3c
  24 3d
  14 3e
  25 3f
  20 40
  19 41
  20 42
  21 43
  19 44
  32 45
  20 46
  18 47
  20 48
  24 49
  26 4a
  21 4b
  17 4c
  16 4d
  25 4e
  18 4f
  25 50
  20 51
  22 52
  18 53
  23 54
  19 55
  24 56
  18 57
  26 58
  26 59
  27 5a
  23 5b
  16 5c
  24 5d
  28 5e
  11 5f
  16 60
  24 61
  28 62
  18 63
  14 64
  30 65
  18 66
  22 67
  27 68
  27 69
  29 6a
  21 6b
  20 6c
  16 6d
  28 6e
  20 6f
  15 70
  25 71
  23 72
  13 73
  18 74
  18 75
  23 76
  17 77
  14 78
  19 79
  21 7a
  23 7b
  24 7c
  18 7d
  20 7e
  29 7f
  18 80
  16 81
  28 82
  28 83
  23 84
  22 85
  18 86
  24 87
  21 88
  26 89
  16 8a
  12 8b
  25 8c
  19 8d
  19 8e
  16 8f
  19 90
  17 91
  21 92
  19 93
  21 94
  19 95
  25 96
  14 97
  19 98
  19 99
  21 9a
  30 9b
  13 9c
  20 9d
  22 9e
  23 9f
  30 a0
  29 a1
  11 a2
  28 a3
  20 a4
  15 a5
  18 a6
  18 a7
  19 a8
  21 a9
  25 aa
  13 ab
  16 ac
  24 ad
  16 ae
  19 af
  17 b0
  11 b1
  16 b2
  14 b3
  24 b4
  15 b5
  23 b6
  17 b7
  20 b8
  24 b9
  11 ba
  21 bb
  27 bc
  21 bd
  26 be
  22 bf
  21 c0
  10 c1
  17 c2
  16 c3
  24 c4
  21 c5
  15 c6
  23 c7
  15 c8
  20 c9
   9 ca
  22 cb
  21 cc
  13 cd
  16 ce
  17 cf
  13 d0
  34 d1
  22 d2
  20 d3
  14 d4
  16 d5
  27 d6
  21 d7
  12 d8
  15 d9
  11 da
  17 db
  21 dc
  21 dd
  20 de
  16 df
  26 e0
  15 e1
  16 e2
  17 e3
  15 e4
  18 e5
  25 e6
  12 e7
  22 e8
  25 e9
  20 ea
  12 eb
  17 ec
  17 ed
  14 ee
  23 ef
  19 f0
  20 f1
  20 f2
  19 f3
  16 f4
  14 f5
  18 f6
  21 f7
  27 f8
  15 f9
  22 fa
  62 fb
  14 fc
  24 fd
  26 fe
  19 ff
  

Also, it might be different on a Master.

Or an Electron? I wonder if this is it. I don't know what the date distribution of the titles in that sub-corpus looks like. One of the tapes that exhibits this phenomenon is Snapper, which was obviously an early game -- the UEF claims to be version 1 -- but I'm sceptical of that claim, since the weird Acornsoft "version files" don't appear in the UEF.

Might it go the other way? I wonder if any of these UEFs were saved on OS 0.1? Or 0.1 with the Richard Russell CFS in-memory patch for the filing system bug? Or even OS 1.0 (which I think did briefly exist)?

Hmm ... don't we have the Acorn source code for MOS somewhere?

[vanekp]

The problem is not with any build UEF files (so not with CSW or makeuef) but the extra CRC bytes appear ONLY when you save something in BeemEm to a UEF image that it adds extra CRC bytes.
If I raw dump a tape file on the BBC there are no extra bytes if I raw dump a uef made in BeemEm then it has the extra CRC bytes in the file.

[SKS1]

Hmm ... don't we have the Acorn source code for MOS somewhere?

Here you go: https://github.com/stardot/AcornOS120

[hoglet]

Hmm ... don't we have the Acorn source code for MOS somewhere?

Here you go: https://github.com/stardot/AcornOS120

Here's the part of the source we have been discussing:
https://github.com/stardot/AcornOS120/b ... MOS74#L665

[Diminished]

The problem is not with any build UEF files (so not with CSW or makeuef) but the extra CRC bytes appear ONLY when you save something in BeemEm to a UEF image that it adds extra CRC bytes.
If I raw dump a tape file on the BBC there are no extra bytes if I raw dump a uef made in BeemEm then it has the extra CRC bytes in the file.

No. It is not that simple.

I've already found real tape dumps that contain 1 extra byte after the CRC -- there is at least 1 of them in your own tape archive:

Hmm ... don't we have the Acorn source code for MOS somewhere?

Here you go: https://github.com/stardot/AcornOS120

Here's the part of the source we have been discussing:
https://github.com/stardot/AcornOS120/b ... MOS74#L665

Thanks ... yeah, I was just looking at this. I was hoping there might be a comment that provided some insight, but there's nothing compelling.

I still want to know why a minority of real-world tapes are leaking a single duplicate CRC byte.

I would try MOS 0.1 if I had a way of getting it into ROM, but I don't.

[hoglet]

I still want to know why a minority of real-world tapes are leaking a single duplicate CRC byte.

I'm still not sure about this either.

All of the critical stuff is occurring during the stop bit (at the end of the second CRC byte), so it's useful to consider the timeline of this bit:

Code: Select all

  0us: beginning of stop bit
416us: ACIA asserts IRQ (Tx Data Empty)
478us: IRQ Handler writes to TxD and ACIA releases IRQ
516us: resetACIA code writes to ACIA Control, setting a Master Reset
832us: end of stop bit

From the distribution of the extra byte values, there isn't much evidence of the ACIA stopping mid-way through the byte. So it's likely that once the next start bit happens, the ACIA is committed to sending the byte.

So for the extra byte to be present, somehow the resetACIA call must be delayed by > 316us.

There is very often another interrupt correlated with the stop bit, and I haven't yet worked out what it is. But it never seems to delay things enough that I see an extra byte being transmitted. I did over a hundred tests in a loop.

Dave

Dave

[hoglet]

From the distribution of the extra byte values, there isn't much evidence of the ACIA stopping mid-way through the byte. So it's likely that once the next start bit happens, the ACIA is committed to sending the byte.

So it appears this is not the case...

I did an experiment where I revectored IRQV1 to a delay loop, allowing me to introduce different amounts of delay. Once the delay exceeds ~400us you start to see the extra byte after the CRC, but it's clear the Master Reset takes effect immediately.

The data on the distribution of the extra bytes does not suggest the are being chopped in this way. So something else must be going on.

Looking at a disassembly of OS 0.10, the code after the CRC is different:

Code: Select all

    JSR .saveChecksumToTape                             ; save checksum to TAPE

.saveFinalTwoBytes
    JSR .saveByteAToTapeInternal                        ;
    INC .fsGotACharacterToReadOrWriteFlag               ;
    JSR .saveByteAToTapeInternal                        ;
    JSR .resetACIA                                      ;

and

Code: Select all

; ***************************************************************************************
.saveByteAToTapeInternal
    LDA #12                                             ;
    STA .verticalSyncCounter                            ; set to 12 vsyncs
-
    BIT .fsGotACharacterToReadOrWriteFlag               ;
    BMI +                                               ; if (byte written) then branch
    BIT .verticalSyncCounter                            ; check vertical sync counter
    BPL -                                               ; if (not timeout) then branch (loop back)
    JSR .checkForEscapeDuringCassetteOperation          ;
    BMI .saveByteAToTapeInternal                        ;

+
    ; byte is now read / written
    PHP                                                 ; store flags
    LDA #0                                              ; A=0
    STA .fsGotACharacterToReadOrWriteFlag               ; clear flag, we no longer have a byte
    PLP                                                 ; recall flags
    RTS 

So I think the behavoiur of this is well worth someone investigating.

Dave

[Coeus]

Dave,

Looking at the Acorn source you linked above, the routine that Toby calls 'saveByteAToTapeInternal' is called IWAIT and does not necessarily cause a byte to be written to tape. It is doing nothing with any of the ACIA regisrters. Instead it just waits for a flag byte, IFLAG, to go negative:

Code: Select all

IWAIT JSR ESCAPE
 BIT IFLAG
 BPL IWAIT

 LDAIM &00
 STA IFLAG ;Zero IFLAG
 LDA IBUF ;Restore A
 RTS  ;Note V preserved

IFLAG is obviously being set in the interrupt service routine. Here is the start of that:

Code: Select all

IRUPT

 DEC IFLAG
 LDA SROMSW ;Is SFS active ?
 BEQ NOTSPK ;Nope

At the point IFLAG is decremented and presumably becomes negative, the routine does not yet know what the cause of the interrupt is. Of course, the code looping on that flag won't see the change until the ISR returns so this could be undone again but that does not seem to be the case during the transmit code path.

As nothing new is loaded into IBUF, I think that means the two calls to IWAIT could cause up to two further copies of the last byte to be retransmitted, provided the source of the interrupt is the TX register becoming empty, but IWAIT will be satisfied by any interrupt from any source and in the case of non-tx-register-empty interrupts, nothing further will be transmitted.

[chrisn]

I've changed BeebEm so that writing a master reset to the ACIA control register stops transmission, and this means BeebEm now writes only one duplicate CRC byte instead of two. I also fixed the UEF output so that the whole file is gzip compressed rather than compressing each UEF chunk separately, which fixes the incompatibility with other emulators.

[Diminished]

So I think the behavoiur of this is well worth someone investigating.

A bit more on a possible OS 0.1 connection with that single repeated byte of CRC.

There are a total of 100 UEF files from Stairway that have a garbage byte that matches the CRC. I was able to find 76 of these on bbcmicro.co.uk, which enabled me to look up their year of release.

Out of 76 UEF files, well over half (45) have a release year of 1982. A further 28 are from 1983. Just three titles have release dates of 1984 or later (and bbcmicro.co.uk might have incorrect dates for one or more of these).

My money is now on the OS 0.1 CFS being responsible for the single-byte leak. hoglet's investigations seem to point quite strongly towards OS 1.2 not leaking anything, and this is backed up by the mass of UEFs that do not exhibit this phenomenon.

Can anyone test it on hardware? All we need is some programs saved using the cassette filing system with an OS 0.1 ROM installed, and then uploaded somewhere as WAV or FLAC. (The OS 0.1 block-zero cassette filing system bug in-memory patch by Richard Russell isn't compulsory for this, since it doesn't really matter if the start of a block gets corrupted.)

[vanekp]

After the comment from Diminished I went though and checked my collection here is a list of the programs that did have it:-

• Adventure200(FoilkadeLtd)1982a 1982 (on every block of the file)
  BeebInvaders(Niblesoft) 1982 (on every block of the file)
  Billiards(HandHsoftware) 1983 (on every block of the file)
  Biology1a(RevisionSoftware) 19?? (only on one block X 0B and only Biology1a)
  Boffin1(Addictive) 1985 (only has it on block 11 Boffin last block)
  CavemanAdventure(ProgramPower) 1983 (on every block of the file)
  Cobra16k RoboSwamp(SoftwareForAll) 1982 (on every block of the file)
  Footer(MicroPower) 1982 (on every block of the file)
  Fortress(Amcom)(B)[TapeSideLabel] 1983 (only has it on one block ? 2D and only SideLabel of tape)
  GeoffCapesStrongman(Martech) 1984 (only on one block GEOFFCAPES 00)
  GrandPrix(SoftwareForAll) 1982 (on every block of the file)
  JR(SoftwareForAll) 1982 (on every block of the file)
  MoonRaider(ProgramPower) 1983 (on every block of the file)
  Robot(Viking Software) 1982 (on every block of the file)
  RowofFour(SoftwareForAll) 1982 (only has it on block ROW-OF-4 00)
  scdump (????) 19?? (on every block of the file)
  sLife-cLife(Golem) 1982 (only has it on block C_life 05)
  Space Hi-way (Amcom) (B) (Tape) [side-smalltabs] 1983 (on every block of the file)
  SpaceHi-Way(Amcom) 1983 (on every block of the file)
  YieArKung~FU(Imagine)(B)[SideLabel] 1985 (only has it on block YIE4 12 last block)
  YieArKung~FU(Imagine)(B)[SideNoLabel] 1985 (has it only on 3 blocks)

some have it on every block some have it only on one or two blocks, I have also put the (c) date that I found in the program if there was one.
This is from a collection of 239 program that I have myself converted either from original tapes or from wav files that where given to my by a forum member.