Major ddos this morning has left many sites (web and mail), inaccessible. The attack is being addressed and seems to be dying down on its own. We apologize for this disruption of service and any updates will be forthcoming.
Our cluster had some file server issues today. Slowing down, even not reachable at times. Everything has been solved by now. Let us know if something not working properly. And sorry for the inconvenience.
The data center will once again be performing more maintenance on the central database beginning Monday, March 27 2007 at 22:00:00 PDT. Downtime window is 6 hours, expected downtime within that window is 4 hours. This will take the following services offline:
Webmail.
All other mail services will remain online, including pop3, imap, and smtp, only the webmail install will be offline. Customer sites and databases will also function as normal.
Because of data base maintenance, the webmail service will be offline starting Monday, March 12 2007 at 22:00:00 PDT. Downtime window is 6 hours, expected downtime within that window is 4 hours. All other mail services will remain online, including pop3, imap, and smtp. Customer sites and databases will also function as normal.
We were just informed one hour ago from our building management that an emergency maintenance window is scheduled for this Saturday morning, February 24th, at 1AM PST. UPDATE! The building has just changed the window to be 23 hours later.. it is now scheduled for Sunday morning, February 25th, at 12:01AM PST. This maintenance window involves taking our entire building’s power offline for approximately 3 hours. The word from the building:>
It was discovered by ABM Engineering during the power
> monitoring
equipment installation, that a Phase C Conductor
> Cable on UPS #4
has been compromised to the point that immediate
> action to
repair it is necessary or a ground fault will occur to
> the building
systems.Since only one of our three data centers is losing power, not all servers will be shut down. Unfortunately, our core routers and our upstream providers are in that data center, so there will be no network to any servers during the window. Gaah.
Since we have advanced (albeit limited) warning of this event, we will be onhand to physically power off all of our equipment at 11:15PM PST. Barring any unforeseen issues (on our end or the building engineer’s) we plan on having everything back up by 4AM PST, hopefully much sooner.
This outage will affect everyone’s sites and email. Email sent to our customers during this outage will be deferred on the sending server and will be delivered after service is restored.
We apologize for the inconvenience this no doubt causes, as well as for the short notice. We will be posting updates as possible during the outage.
Connection to one of our file storage servers is causing inability to check mail for some of our customer (only one group of servers -cluster- is affected). Example: ERROR: Connection dropped by IMAP server in webmail.
Has been fixed at 1:00pm Pacific.
Sorry for the inconvenience.
Resolved outage on mail clusters:
A network hiccup caused our load balancers to reset all connections to the backend authentication databases, which in turn caused all mail servers to need a reset to get their connections back.
Tonight, Thursday, Jan. 18th, at midnight CST, will open a timeframe of two hours in which all our webmail services will be down. So from 0:00CST to 2:00CST, Jan 19 2k7, you will experience 1 hour webmail down. The POP3 mail won’t be affected.
Sleep well, sweet dreams, ignore this while sleeping.
Here’s why some Earthlink customers were having issues with emails:
CBL temporarily inhibited our IPs from being blocked several days ago, so our listings there (and with XBL/spamhaus) have ceased. We’re still talking with them to get this resolved permanently, and our IPs should not be blocked by them in the meantime. As many of you noticed, however, mail to Earthlink is still being blocked.
We have been emailing Earthlink since 12/30, before we ever contacted CBL. After 6 emails we unfortunately haven’t received anything but autoresponses from Earthlink’s blacklist/abuse address. We’ve spoken with their techs via chat and phone and have not been able to get through to anyone above their first tier of support; attempts to transfer to a supervisor were met with 20 minutes on hold and the call being dumped.
We will still be attempting to contact Earthlink to get this particular issue resolved, but if you are an Earthlink customer and are being impacted by this (such as if you utilize forwarding to an Earthlink address — and we know there are several thousand of you), you may wish to contact their support and let them know that we have been writing them to get information about this block and how it can be resolved. [...]
Finally, I do apologize for the impact on everyone. Blacklistings are no fun for anyone, and unfortunately we’ve found that large ISPs are the biggest headache to get de-listed from, as it’s incredibly hard for us to get a human response out of them. Still, we can’t fix the problem on our end (if there is one) without them responding to us to let us know what the problem is. There needs to be a reasonable amount of earnest communication on both ends for these issues to be resolved, and we are certainly trying to meet our end of the bargain there! We aren’t shy about disabling spammers when we find them, and certifiable spammers are actually pretty rare on our primary SMTP machines judging by the (few) complaints we get.
Thanks to everyone for continued patience, which understandably may be wearing thin!
Since long no alert, was about time…
We are currently experiencing networking issues.
Some of you may have been noticing some problems with your mail and MySQL services (and possibly some others). We’re not 100% sure what is causing these network problems, or what is being affected, but our entire admin team is currently investigating the issue. Stay tuned, we’ll be updating this post once we have any new information.
# The tsunami passed over our moonkah.net ‘beach’ about an hour ago. As writing this all moonkah.net services are up and running OK.
Oh, and Merry Christmas! I hope not to post another alert here before Valentine’s.
Georg