We are back... well sort of...

Well as most of you are aware we suffered an outage last Friday which has lead to some extensive downtime and despite bringing most of our sites online we still have some way to go.



What happened?

Simply put, we were the victims of a ransomware attack. This was not a targetted attack and was the result of a combination of a couple of variables, some outside of our control.

We believe the attack happened laterally in otherwords one of our hosting providers customers was already compromised and whilst we were protected from external attacks we werent as well protected from an internal one due to limitations around how we managed our server.


Is my data safe?

Yes. The Habbox sites operated in a virtualised environment, the ransomware encrypted the virtual machines we run within this environment.

There is no evidence that the virtual machines themselves were compromised, looking at the egress traffic logs it was apparent that the virtual machines were never exported from the server. Our virtual machines were ~750GB in size and our monitoring dashboard showed a total data egress of approximately 3GB, this month prior to the attack which aligns with normal expected utilisation.

We identified the ransomware within 5 mins of the sites going down and booted the server into recovery mode thus cutting off the attackers access.



How did you recover?

Recovery took longer than expected in part as everyone was busy (downtime seldom comes at a convenient time). Ultimately I came to the decision that it was time to separate from our former hosting provider and instead opt for a new hosting architecture for the Habbox sites. This includes a combination of SaaS and PaaS services which we are still adopting. This was something I was considering in the months leading to this attack but hadn't realy made any progression with until the recent urgency.

Our databases which were backed up daily have been restored from the night before we went down, our site files have been restored from the previous month which may have resulted in a few broken images around the sites.


What is next?

We have a lot to do in terms of hardening and restoring some of our backend services which includes the radio. We will be working on this over the next few days.