Forum Software Upgrade

feedback, comments and suggestions pertaining to the *. forums
User avatar
MartinB
Posts: 4511
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity

Forum Software Upgrade

Postby MartinB » Sun May 15, 2016 3:14 pm

It didn't bode well, the down-time announcement began...

"We are done for......"

:shock:

:lol:

User avatar
MartinB
Posts: 4511
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity

Re: Forum Software Upgrade

Postby MartinB » Sun May 15, 2016 3:24 pm

...and it doesn't now seem to want to display a picture when I upload it :(

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 3:33 pm

Bums.

d.

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 3:33 pm

Actually, I know what the problem is...

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 3:47 pm

Should work now.

d.

User avatar
1024MAK
Posts: 6676
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...

Re: Forum Software Upgrade

Postby 1024MAK » Sun May 15, 2016 4:11 pm

Image

Mark
Attachments
StarDot Approved-by-the-dark-side.jpg
(32.35 KiB) Not downloaded yet
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk NOW!
BeebWiki‬ - for answers to many questions...

User avatar
MartinB
Posts: 4511
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity

Re: Forum Software Upgrade

Postby MartinB » Sun May 15, 2016 4:20 pm

Is everything deffo ok coz Dave Hitchins' sig looks broken....? :-k

User avatar
1024MAK
Posts: 6676
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...

Re: Forum Software Upgrade

Postby 1024MAK » Sun May 15, 2016 4:23 pm

Roland wrote:We tried to figure out why this happened but it took too long so we decided to restore the old files and database. This restore didn't work via the web interface so we restored it on the command line. To make that work we had to increase the max_allowed_packet value in the /var/db/mysql/my.cnf file. After that all seems rights except DaveH's signature....]

And other (I presume unicode) text, like the Euro symbol € in this post :(
Zarchos wrote:Aren't you French ? Or at least living in France ?
I've been to your site and well, seeing a Minitel and prices given in € euros ...


Mark


Edited by Roland to test the € sign.
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk NOW!
BeebWiki‬ - for answers to many questions...

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 4:24 pm

Curses, it's done something to the DB table encoding... *shakes fist at sky*

d.

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 6:21 pm

OK - Other than euro symbols and DaveH's signature, has anyone noticed any other characters in a pickle?

d.

User avatar
BigEd
Posts: 1392
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Forum Software Upgrade

Postby BigEd » Sun May 15, 2016 6:36 pm

There's some unicode(?) in this very recent post which looks all wrong:
viewtopic.php?p=137833#p137833

User avatar
1024MAK
Posts: 6676
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...

Re: Forum Software Upgrade

Postby 1024MAK » Sun May 15, 2016 6:47 pm

That's very wrong. Looks like I have let the dog do the typing :lol:

Mark
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk NOW!
BeebWiki‬ - for answers to many questions...

User avatar
1024MAK
Posts: 6676
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...

Re: Forum Software Upgrade

Postby 1024MAK » Sun May 15, 2016 7:08 pm

In front of £ (UK pound) signs here.
One here
At this point, I was thinking it was a Dave problem :lol:
But no, Wim's posts as well, like here

:(

Mark
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk NOW!
BeebWiki‬ - for answers to many questions...

User avatar
MartinB
Posts: 4511
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity

Re: Forum Software Upgrade

Postby MartinB » Sun May 15, 2016 7:17 pm

And yet the pound sign I posted today was fine. Again look..... £ £ £ £

(but then again I don't have a scoobies about what's going on with this stuff..... :- )

User avatar
hoglet
Posts: 6359
Joined: Sat Oct 13, 2012 6:21 pm
Location: Bristol

Re: Forum Software Upgrade

Postby hoglet » Sun May 15, 2016 7:18 pm

Hi Daniel,

You probably know this already....

In the example Ed posted above:
viewtopic.php?p=137833#p137833
it seems that a unicode "RIGHT SINGLE QUOTATION MARK" character (U+2019) was included in the original post.

It looks like this was originally saved into the database using a UTF-8 encoding:
0xE2 0x80 0x99

Since the update, that sequence is now displayed as three separate characters, using the Windows CP1252 character set:
https://en.wikipedia.org/wiki/Windows-1 ... age_layout

This seems to suggest that the update reverted MySQL's default character set from UTF-8 back to latin1 (which is almost the same as CP1252)

Dave

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 7:27 pm

This is rather irksome, as it was just a dump of all the tables which were read back in after we realised that 3.1.9 was not playing ball...

d.

User avatar
BigEd
Posts: 1392
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Forum Software Upgrade

Postby BigEd » Sun May 15, 2016 7:37 pm

Is there a single global tweak which would fix all the old posts, at the risk of slightly breaking the few non-ascii posts made since the restore?

I've wrestled with this kind of thing in a past life, but I don't feel I can offer any advice with confidence. I can confirm that it's irksome!

User avatar
Arcadian
Posts: 2733
Joined: Fri Nov 24, 2000 12:16 pm
Contact:

Re: Forum Software Upgrade

Postby Arcadian » Sun May 15, 2016 7:44 pm

Hmm ... my avatar seems to have disappeared ... will be easy for me to fix, but I was wondering if anybody else has been effected by this? (From what I can tell - no!).
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk

Image
ABug Leicestershire (17-19 November 2017)

User avatar
roland
Posts: 2778
Joined: Thu Aug 29, 2013 8:29 pm
Location: Born (NL)
Contact:

Re: Forum Software Upgrade

Postby roland » Sun May 15, 2016 7:58 pm

I installed a test server with the forum and used the database from the backup. Same problem. We will have to look for a method to fix the broken posts because if we're doing a new restore that would fix the issue (don't ask me how at the moment) we will loose all new posts since the forum came back online. I don't think we want that. There are almost 50 posts since then....
256K + 6502 Inside
MAN WOMAN :shock:

User avatar
hoglet
Posts: 6359
Joined: Sat Oct 13, 2012 6:21 pm
Location: Bristol

Re: Forum Software Upgrade

Postby hoglet » Sun May 15, 2016 8:06 pm

Roland,

Is it possible to find that that post referenced above in the MySQL backup?

Maybe by greping for "I am very interested in this project, looking forward to see how you get on."

That would let us see whether the backup is somehow messed up (which I doubt).

It still seems most likely that the database is fine, but the restored forum software is somehow specifying the wrong character encoding when connecting.

Dave

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 8:21 pm

Probably the easiest thing is to just run an update query for the rogue characters along the lines of

Code: Select all

UPDATE thingytable SET column = REPLACE(column, 'roguecharacters', 'propercharacter')


Unless anyone can think of a reason why that wouldn't work?

User avatar
roland
Posts: 2778
Joined: Thu Aug 29, 2013 8:29 pm
Location: Born (NL)
Contact:

Re: Forum Software Upgrade

Postby roland » Sun May 15, 2016 8:23 pm

Indeed, the database looks fine on the console:

Schermafbeelding 2016-05-15 om 22.15.46.png


But new posts are messed up in the database:

Schermafbeelding 2016-05-15 om 22.19.46.png



So I think it's best not to use any special characters until we solved this issue.
256K + 6502 Inside
MAN WOMAN :shock:

User avatar
BigEd
Posts: 1392
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Forum Software Upgrade

Postby BigEd » Sun May 15, 2016 8:26 pm

danielj wrote:Probably the easiest thing is to just run an update query for the rogue characters along the lines of

Code: Select all

UPDATE thingytable SET column = REPLACE(column, 'roguecharacters', 'propercharacter')


Unless anyone can think of a reason why that wouldn't work?



Depends on whether there's a latin-1 equivalent for all the characters people have used... it doesn't seem like an ideal solution to me.

If you were able to redo the import, have you no way to replay the subsequent updates into the result, to bring it up to date?

Dave's idea seems hopeful: the database is fine but the forum software is not reading it right. There's a configuration tweak needed.

User avatar
danielj
Posts: 5133
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Forum Software Upgrade

Postby danielj » Sun May 15, 2016 8:28 pm

It never rains... :oops:

User avatar
roland
Posts: 2778
Joined: Thu Aug 29, 2013 8:29 pm
Location: Born (NL)
Contact:

Re: Forum Software Upgrade

Postby roland » Sun May 15, 2016 8:56 pm

I subscribed to the PHPBB support forum and posted this issue there. The post has to be confirmed by an admin so I have to wait a little while before I can expect an answer. Let's keep calm, don't over rush things and stop using £ , € and other special characters for the time being.
256K + 6502 Inside
MAN WOMAN :shock:

User avatar
hoglet
Posts: 6359
Joined: Sat Oct 13, 2012 6:21 pm
Location: Bristol

Re: Forum Software Upgrade

Postby hoglet » Sun May 15, 2016 9:42 pm

Roland,

Can you grep for "charset collation" in the backup?

Dave

User avatar
1024MAK
Posts: 6676
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...

Re: Forum Software Upgrade

Postby 1024MAK » Sun May 15, 2016 10:28 pm

Me 'alf's have been abused :shock: (link)

Mark
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk NOW!
BeebWiki‬ - for answers to many questions...

User avatar
paulv
Posts: 3546
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Forum Software Upgrade

Postby paulv » Mon May 16, 2016 2:28 am

hoglet wrote:Roland,

Can you grep for "charset collation" in the backup?

Dave


Collation and encoding in a dB are related but not the same thing typically. In SQL server you can choose Latin collation but it uses UTF-16 for its encoding of data regardless of what the collation is.

Essentially collation controls how result sets are ordered, encoding sets how characters are stored.

The issue like like a classic mismatch in encoding between the site and the dB. Collation should have no effect on this. Typically this sort of issue occurs when the dB stores utf-8 encoded data (in utf-16 internally) and the website presents it with an encoding of windows-1252 or ISO-8859-x where in the UK, x is typically 1.

ISO-8859-1 is meant to be a subset of utf-8 but there are subtle differences and the symptoms we're seeing are indicative of and encoding mismatch as I've described. I'd be checking all the site config files and dB driver config settings to make sure they're set to match the dB as it has been restored and changing those settings if going to be mismatched.

Paul

paulb
Posts: 762
Joined: Mon Jan 20, 2014 9:02 pm

Re: Forum Software Upgrade

Postby paulb » Mon May 16, 2016 10:09 am

paulv wrote:
hoglet wrote:Roland,

Can you grep for "charset collation" in the backup?

Dave


Collation and encoding in a dB are related but not the same thing typically. In SQL server you can choose Latin collation but it uses UTF-16 for its encoding of data regardless of what the collation is.

Essentially collation controls how result sets are ordered, encoding sets how characters are stored.


I guess Dave was referring to various keywords MySQL uses that might indicate database- or table-level settings in a database dump. It's been a while since I used MySQL - I tend to choose PostgreSQL instead, whose locale/charset/collation properties are a bit more rigid than MySQL's "configurable everything" - so I can't really name the settings to look at without looking at the horrible MySQL documentation for longer than I can bear.

User avatar
roland
Posts: 2778
Joined: Thu Aug 29, 2013 8:29 pm
Location: Born (NL)
Contact:

Re: Forum Software Upgrade

Postby roland » Mon May 16, 2016 12:20 pm

Hi guys,

I probably have a solution \:D/

On my test system I did a restore via PHPBB. If it makes the backup it will probably restore it well, at least that was what I thought. Yesterday I didn't see what went wrong but now that I'm having my own system to play with, I can do whatever I want without being afraid to mess up something.

Screen Shot 2016-05-16 at 14.14.48.png


If I restore now we will loose 24 hours of posting. So I want to takes things real easy. I start with making a new backup of the current database and write a small script to copy the new posts to my test database. If that runs fine I will do this trick on the life database. In the end we might end up with a decent forum with all special characters restored. But for now, please try to avoid them as much as possible.
256K + 6502 Inside
MAN WOMAN :shock:


Return to “stardot.org.uk”

Who is online

Users browsing this forum: No registered users and 2 guests