Whats going on again

[scottish]

http://stats.pingdom.com/0rmh1vp1dnyh/410394

Down to 97% uptime, forums constantly slow or dead

http://stats.pingdom.com/0rmh1vp1dnyh/410394/2011/12

been going down everyday again.

[GoldenMerc]

Yeh was down loads earlier

[Catzsy]

http://stats.pingdom.com/0rmh1vp1dnyh/410394

Down to 97% uptime, forums constantly slow or dead

http://stats.pingdom.com/0rmh1vp1dnyh/410394/2011/12



been going down everyday again.

I have to agree with you here even though it pains me to criticise but I am sure we are losing members because of the constant lag/not loading. It is not as if members don't constantly report it. I hope it is sorted soon as a very busy xmas hols will be on us shortly :(

[Recursion]

It's down at least once a day. Yesterday I opened 5 threads in different tabs with the intent to reply but the forum died and I gave up making the posts.

[Chippiewill]

It looks as if the problem has resolved itself now, seems likely it was an isolated incident.

Edit: Nevermind, interesting pattern going on here:
http://stats.pingdom.com/0rmh1vp1dnyh/410394/2011/12

Seems to coincide with DB backups?

Edit2: Looking at the other months, it seems that it happens once every day at 2:25-2:35 and then again at a time which moves forwards by a few minutes each day. Probably something which happens every 24 hours after it previously finished.

[scottish]

The early morning ones are the db backups

The ones later on in the day (5th at 10, 11 12 and 1, today at 11 until 1, etc) god knows.

[Chippiewill]

The one later in the day last almost exactly an hour and a half (Removed the single minute downtimes in testing this as they're likely increased traffic from after downtime) and move around by about an hour every ten days.

[beth]

the last week or so it's been a bit of a joke considering we're on the fancy new server now.

[Chris]

It's been a bit of a pain, I was trying to read threads earlier today but gave up in the end. :(

[HotelUser]

According to that link it seems that it's quite slow to connect to in Canada, what a shame :P

I haven't noticed much significant downtime other than when the server undergoes its daily backups (1-2am for most of you folks). Would I be correct in saying the downtime caused by this is primarily the downtime we're talking about here? It might be worth bringing up the idea to have server backups changed to start at 1-2am to 5am.

[scottish]

No, the downtime i'm talking about is the later downtime

i.e. today at 11am until 1pm our time the forum was down majority of the time, and took 5 mins to load for the rest of the time.

[Chippiewill]

According to that link it seems that it's quite slow to connect to in Canada, what a shame :P

I haven't noticed much significant downtime other than when the server undergoes its daily backups (1-2am for most of you folks). Would I be correct in saying the downtime caused by this is primarily the downtime we're talking about here? It might be worth bringing up the idea to have server backups changed to start at 1-2am to 5am.
Glad you bothered reading the thread... the backups are at 2:30 AM and we have a cycling downtime of length hour and a half changing time each day by moving on average six minutes forward each day.

[Eoin247]

Didn't notice it myself yet. It's certainly not as bad as last years problem though

[Recursion]

According to that link it seems that it's quite slow to connect to in Canada, what a shame :P

I haven't noticed much significant downtime other than when the server undergoes its daily backups (1-2am for most of you folks). Would I be correct in saying the downtime caused by this is primarily the downtime we're talking about here? It might be worth bringing up the idea to have server backups changed to start at 1-2am to 5am.

I know plenty of forums, larger than Habbox, that manages database backups without the entire server committing suicide in some form...

[Catzsy]

According to that link it seems that it's quite slow to connect to in Canada, what a shame :P

I haven't noticed much significant downtime other than when the server undergoes its daily backups (1-2am for most of you folks). Would I be correct in saying the downtime caused by this is primarily the downtime we're talking about here? It might be worth bringing up the idea to have server backups changed to start at 1-2am to 5am.

Well the weird thing is David it started a few weeks ago every day between 7am and 9am then seemed to get later between 9am and 11am and today it was an hour later again. We are talking about every day and it can last up to two hours. Very frustrating. I would say back ups during the night would be a lot better. Unfortunate for some international members but better for the majority who post during the day. :)

[xxMATTGxx]

I know there was a deadlock in the database the other day which caused problems for Habbox itself. I don't have much else to say but I'm aware there is a few problems during the UK time at least and it is quite annoying since it's when a lot of the members would be online. I'll keep telling Jin about the problems we are having and hopefully get a solution in place soon.

[Recursion]

I know there was a deadlock in the database the other day which caused problems for Habbox itself. I don't have much else to say but I'm aware there is a few problems during the UK time at least and it is quite annoying since it's when a lot of the members would be online. I'll keep telling Jin about the problems we are having and hopefully get a solution in place soon.

Deja vu... ;)

[Chippiewill]

I know there was a deadlock in the database the other day which caused problems for Habbox itself.Clearly they're happening for a regular reason, they're VERY predictable. So if it is a database lock it's locking every 24 hours or due to a function which occurs every 24 hours.

[Recursion]

I still don't understand why Jin doesn't hire a second admin that he trusts who can be around when he isn't, and who can modify configs etc to make it run more stable generally (i.e. not dying just because a database is being backed up).

[Chippiewill]

Jin is around at the moment, he just hasn't noticed yet, give him time.

[GeorginaxD]

I noticed this morning / afternoon I couldn't get on the forum.

[Eoin247]

I felt it this time, was pretty bad there

[HotelUser]

Glad you bothered reading the thread... the backups are at 2:30 AM and we have a cycling downtime of length hour and a half changing time each day by moving on average six minutes forward each day.

You're welcome :)


I know plenty of forums, larger than Habbox, that manages database backups without the entire server committing suicide in some form...

Probably, so if the downtime is from our backups (note that the server also backs up files not just the database), then it could just be doing it in an inefficient manner. I'm not sure about the fine details on the backup script the server runs, but it's possible it's simply horrible inefficient, I do not know.


Well the weird thing is David it started a few weeks ago every day between 7am and 9am then seemed to get later between 9am and 11am and today it was an hour later again. We are talking about every day and it can last up to two hours. Very frustrating. I would say back ups during the night would be a lot better. Unfortunate for some international members but better for the majority who post during the day. :)

Oh I see now, that would explain why I've never noticed because I'm fast asleep at 7-9AM GMT (3-5AM here). If these downtimes are every day and following a pattern of being in the mornings then it could be some sort of backup running, or perhaps another synchronized task running each day on the server. Whatever the reason we can only hope that Jin will be able to pop on and resolve the pr

[Mathew]

I still don't understand why Jin doesn't hire a second admin that he trusts who can be around when he isn't, and who can modify configs etc to make it run more stable generally (i.e. not dying just because a database is being backed up).
You'd think that after the site was down for a few days, it might have been some form of catalyst towards asking someone else to join Jin as a second admin. Sadly, it appears that wasn't the case and we're in a similar situation where the poor old GM team seem helpless. We are relying on one person who is away for a significantly large proportion of the year - it is a completely flawed way to run such a popular forum and I really do hope some change to the structure is brought about.

Above all though, it's the Christmas holidays and it's one of the busiest times of the year for activity on Habbox. Obviously I can only speak on behalf of the Events Department, but our organisation is largely based on the forum and it makes our jobs extremely difficult when we can't update things through this medium. Come on, we encourage Events Organisers to advertise HxF in their events.. :P.

[Eoin247]

As Mathew said, another admin would be a great help. Even if this new admin just came on part time, it could be the difference between frustrating members into leaving/not joining and having a more active forum.

[xxMATTGxx]

The other admin would only help if Jin wasn't around. Although that's not really up to me but anyway, I do believe Jin has been speaking to the datacenter about a problem we have been facing recently and hopefully we can get that fixed pretty soon as well. I know he was also looking into another thing which was causing us problems as well.

[scottish]

then it goes down for another hour or so :P

http://stats.pingdom.com/0rmh1vp1dnyh/410394/2011/12

[Stephen]

ever thought jin might be a jin-x? maybe we should send him up on a rocket on nye then get someone else willing to pay to take his place

[HotelUser]

This is now a pretty serious issue and it's going to be addressed.

[The Don]

I'm using internet explorer right now and I can't see any problems nor any lag

[Chippiewill]

This is now a pretty serious issue and it's going to be addressed.
It's always been an issue. As I've said in the past the downtime is moving around day by day and it's now gone into peak time and suddenly people are noticing it again. Previous times people have noticed are when it's been at the ~12PM peak time and the 9AM peak time.

[Chippiewill]

It seems as though the server problems have been fixed [/sarcasm]

[GoldenMerc]

Server needs to be thrown into the bin

[sophiethenerd]

It started like this last year. We do not want downtime like it was last year.

[Stephen]

wait so you only address these issues when they're serious? If you did something when the server started to **** up a few weeks ago then maybe it wouldn't become a serious issue

[HotelUser]

It's always been an issue. As I've said in the past the downtime is moving around day by day and it's now gone into peak time and suddenly people are noticing it again. Previous times people have noticed are when it's been at the ~12PM peak time and the 9AM peak time.

It was a serious issue previously in the year yes and again now our downtime is beginning to be abit... much :P Hopefully shortly Jin will be able to lay down the lay with the server and/or datacenter and bring our uptime back upto something more reasonable and consistant.

[Chippiewill]

It was a serious issue previously in the year yes and again now our downtime is beginning to be abit... much :P Hopefully shortly Jin will be able to lay down the lay with the server and/or datacenter and bring our uptime back upto something more reasonable and consistant.
If you look on pingdom you can see the downtime happens every day but is just moving round through the day (Roughly 6 minutes per day), presumably when it hits a peaktime the website struggles, goes down then everyone spams refresh causing the webpage to stay down. It could the the datacentre or it could be the habbox server, either way the problem was NEVER solved it just became less pronounced.

[HotelUser]

If you look on pingdom you can see the downtime happens every day but is just moving round through the day (Roughly 6 minutes per day), presumably when it hits a peaktime the website struggles, goes down then everyone spams refresh causing the webpage to stay down. It could the the datacentre or it could be the habbox server, either way the problem was NEVER solved it just became less pronounced.

The previous problem was one on our old server, it was resolved when we moved onto the new server but now there's just another problem giving us abit of hell.

[GoldenMerc]

It defiantly needs sorted soon though

[Recursion]

When it comes to server stability, you need to be proactive, not reactive. Evidently the problem wasn't solved, it was just pushed back into the night (or "swept under the carpet")


The previous problem was one on our old server, it was resolved when we moved onto the new server but now there's just another problem giving us abit of hell.

The common factors here then are:
a) Jin
b) the configuration Jin uses

[HotelUser]

When it comes to server stability, you need to be proactive, not reactive. Evidently the problem wasn't solved, it was just pushed back into the night (or "swept under the carpet")



The common factors here then are:
a) Jin
b) the configuration Jin uses

I am unsure as to whether or not there are multiple causes however I am aware that the datacenter was contacted the other day because there was a problem on their end. Whatever the case I'm not disagreeing with you or anyone else that the recent downtime has become a problem which needs resolving.

[HarrySX]

I am unsure as to whether or not there are multiple causes however I am aware that the datacenter was contacted the other day because there was a problem on their end. Whatever the case I'm not disagreeing with you or anyone else that the recent downtime has become a problem which needs resolving.

This is UK2's London dc you're talking about right? Best of luck playing ping pong trying to communicate with their Indian support, you'll need it.

The problem will go on and on and Habbox's user activity will decrease.

[Recursion]

Looks like they're with 100tb.com (querying the Ripe DB)? So yeah, UK2.

[HarrySX]

Looks like they're with 100tb.com (querying the Ripe DB)?

Probably because of the sheer amount of bandwidth Habbox use. 100TB are in UK2's London datacentre which is shoddy to say the least.

100TB's SoftLayer services areepiphenomenonbut the London service is terrible.

[Showder]

GMT+8 timezones , the forum usually works well but when I was on UK time few days ago due to christmas, boy was it long to load