Followup

September 19th, 2006 - No Responses

Looks like the dreaded summer of network glitches coming to an end:

It’s been a few days since our last change to our network and things are still looking very good. There are some isolated (and seemingly unrelated) issues we are dealing with, but the major network issue does appear to be resolved at this time.

This latest change has left our network setup in a not fully redundant state and we are working as fast as possible to fix that.

Additionally we are working to improve our power redundancy for our core network equipment, reorganize our network to provide more long-term scalability and improve performance, and further optimize the configurations on the new equipment we have already installed. Once everything is done we will have invested somewhere in the neighborhood of $300,000 on upgrades and improvements to our network, not to mention all of the human time involved. This is our highest priority.

Router Maintenance Complete

September 12th, 2006 - No Responses

Things are up again and running like never before:

We are confident that all the minor issues following the router hardware swap this evening are now resolved. If you are having any issues with your site or email please contact our Tech Support team.
While initial performance looks promising we’ll need another 12-24hours before we will have concrete evidence that the hardware replacement has resolved the major issues.

Monday Downtime, not Sunday (rectifying)

September 9th, 2006 - No Responses

Oops, downtime will be on Monday night:

We’re rescheduling the network maintenance previously scheduled for 8pm PDT Sunday night (Sep. 10) to 10pm PDT Monday the 11th so that we can perform a more thorough upgrade. We will be taking the network down for some fairly invasive central network upgrades, so most everything will be down for the duration. We’re now expecting downtime to be in the range of 30-45 minutes due to the extra work, but doing things this way will prevent a second or third outage later in the week. During this time everything (web, ftp, email, panel) will be unavailable.

Network downtime Sunday night

September 7th, 2006 - No Responses

Hope you’re gonna sleep during those minutes, ain’t ya?

At 8pm PDT Sunday night (Sep. 10) we will be taking the network down for some fairly invasive central network upgrades. We don’t expect downtime to exceed 20 minutes, but during this time everything (web, ftp, email, panel) will be unavailable.

Joomla CMS 1.0.11 Upgrades

September 7th, 2006 - No Responses

For all of you running a Joomla CMS website with us: just upgraded the engine to the latest 1.0.11 security release. Some thrid party extensions may (just may) not work properly under the new tights…

If noticing something, give me a line.

Thanks,

Georg

Slow mail traffic, and sometimes web

September 5th, 2006 - No Responses

If you’re noticing late mails and slow sites, here’s the reason why:

One of our network storage servers, aka filer, went down this morning. The admins are working on getting it working again. This is only affecting a very small percentage of our customers (we have a multitude of filers in service), and it means that web and mail servers that are interacting with it are unable to pull files, and hence sites are down or email access is limited. Some servers are (not exclusive list): nash, bundy, alvarado, snicker, and parts of the spunky mail cluster.

We apologize for the inconvenience, and will be posting updates as soon as possible.

— Things should be back to normal as editting this post.

Hot and Cool

August 31st, 2006 - No Responses

Alright, we start with the good news and then plunge (a bit) with the bad ones — not really bad for those alien to @comcast.net who can majestically ignore the bad news and enjoy.

Good news first:

After many more incremental improvements and configuration changes we’ve decided to upgrade our core router. The upgrade will increase CPU power as well as total throughput which should definitely alleviate many if not all of the problems we’ve been seeing lately.

The upgrade should be fairly seamless but we will let you know ahead of time if we anticipate downtime.

Hopefully this will help with all the NFS problems that have been affecting site and email slowness.

Comcast forwards to be disabled:

We’ve been contacted by someone very helpful at AOL and I think we have that problem squared away, at least for now. There have been no further blocks and the AOL contact has indicated that our thresholds for blocking are much higher now. We’re waiting for a bit to see whether this will solve the problem long-term, and are looking at implementing some suggestions they have made in the meantime to ensure that we can hopefully stay on their good side. We still won’t be allowing new AOL forwards or forwards that have been removed to be re-added, but the existing ones won’t be disabled for the time being. I still recommend setting up a local mailbox rather than forwarding any mail if possible, as forwards of any type add a potential failure point in your email’s path.

Now, for the bad news — Comcast has become an increasing problem in the last two weeks and is now completely denying our unblock requests. As a result:

In 7 days, on Wednesday, September 6th, we will be disabling all forwards to @comcast.net addresses.

As a bit of background:

Comcast blocks are atypical from the others that we’ve been having problems with in that they last indefinitely until unblocked manually. Unlike AOL blocks (which phase out automatically after 24 hours - though may reappear) someone has to flip a switch over there for any future mail to go through. The unfortunate part is that they have zero human availability and all we get from their blacklist email address are auto-responses — either the IP is automatically unblocked, or the unblock is denied and the phone number of their abuse/security department is given. Unfortunately, this phone number is a completely unmanned voicemail drop-box. We’ve left no less than six messages on their voicemail in the last couple months, and despite numerous requests we have never received a phone call back or an email response. We’re not even asking that they remove the blacklist — we’re simply asking for more information on why the IPs were blocked, and for a sampling of the typical spam they are supposedly getting from us! In fact, the only response we can get from them (if we get one at all) is an automated form message saying that “most of the email” received from our IPs is spam, which we know, in fact, is false.

Again we regret that this decision had to be made, but we’re currently wasting a great deal of time answering complaints regarding Comcast blacklists, not to mention calling and emailing Comcast to have those blacklists removed (unsucessfully) — and they are a very small fraction of our total email forwards. For reference, our number of Comcast forwards is 1/12 of our GMail forwards, and GMail gives us zero problems while somehow managing to filter mail very well for its user base.

For the people who have Comcast forwards set up, I recommend that you remove them yourself and set up a local DreamHost user where mail can be downloaded via client software or checked via webmail. You can edit the destination of an email address by going to “Mail” -> “Manage Email” on the left-hand side in the DreamHost control panel, and click on “Edit” next to the email address in question. If you have difficulty with this, please contact support.

Next week, for those addresses that don’t have an alternate recipient other than a Comcast forward, we will create a local email account for mail to be routed to, so that nothing is lost.

AOL forwards to be disabled

August 22nd, 2006 - No Responses

From the hardware bods (Important for any @aol.com fans; anyone else may disregard):

As per AOL’s request of us, all forwards from DreamHost-hosted addresses to AOL accounts will be disabled in one week, on Tuesday, August 29th.

Some background on the problem: We’ve been having intermittent problems with automated AOL volume and spam blocks since February, after our mail system was load-balanced. We’ve also grown a great deal during this time, which has added to the general mail volume passing through our servers and contributed to the problem. The load-balancing served well to even out internal server loads and mail queues, but has not worked out so well for our external relationship with other mail providers like AOL, since we now have more machines connecting on fewer IPs.

To address spam complaints, we set up a feedback loop with AOL that sends any emails generating spam complaints back to us. The servers that send original mail receive generally 0-2 complaints per day, while the MX servers (that receive mail, and only send for forwards/autoresponders/bounces) are getting 10-30+ complaints each. Looking at the individual complaints: almost all of it is due to forwarded mail.

Complaints from forwarded mail happen when someone sends mail to a DreamHost-hosted email address, which in turn is set up to forward mail to an AOL address, and the recipient on AOL’s end clicks “This is Spam”. AOL counts this as a complaint against us, even though the mail just passed through our system and did not originate with us. We’ve been trying to address this by warning and disabling AOL forwards for individual accounts as appropriate, but at this point, this is getting impractical to deal with, and we’re not seeing any less AOL complaints despite the effort (because new ones continue to be added).

We’ve called AOL’s postmaster helpdesk numerous times since February and are only able to reach their first level technicians who ask us to either a) submit another whitelist request, or b) write an email, and then c) call back later. Emails directly to the postmaster and whitelist requests explaining how our mail system works and asking for clarification on their policies regarding forwaded mail have been ignored or replied to with a form email that did not answer the questions asked. Essentially, we’re unable to get an actual human response from whomever is making the decisions over there.

In my most recent conversation with an AOL postmaster technician on the phone I was told directly that we should disable AOL forwards, as they will not be whitelisting our IPs while they are generating complaints, and they will not be changing their complaint logic to only target the original spammers.

We regret that we have to make this change and it’s something that we’ve been trying to avoid, but we can’t continue to allow our customers who are trying to send new mail to have problems due to these forwards, and we can’t continue spending copious man hours attempting to manage a problem caused by a single ISP, no matter how large they may be. Finally, AOL’s own representative has asked that we stop forwarding mail to AOL — it can’t get much simpler than that.

For people who have AOL forwards set up, we recommend that you remove them yourself and point them at a non-blocking provider that still offers spam filtering (like GMail) or have it go to a local DreamHost user where it can be downloaded via an email client or checked via webmail. You can edit the destination of an email address by going to “Mail” -> “Manage Email” on the left-hand side in the DreamHost control panel, and click on “Edit” next to the email address in question. If you have difficulty with this, please contact support.

Next week, for those addresses that don’t have an alternate recipient other than an AOL forward, we will create a local email account for mail to be routed to, so that nothing is lost.

Issues around us

August 21st, 2006 - No Responses

We are experiencing some networking issues this morning that is causing certain web servers to crash and is slowing down portion of the mail delivery. Our admins are looking into it, and we’ll post an update as soon as we have new information.

Our Moonkah.net hosting zone looks still clear of damage. Let’s hope it’ll remain so.

Relay Access Denied

August 15th, 2006 - No Responses

If you’re receiving Relay Access Denied errors after trying to send mail in your email client, please try the following before you contact support. This applies even if you haven’t touched your settings and it’s been working before:

Make sure that you have SMTP authentication enabled and using the same login that you have set for incoming mail (check your Outgoing Server settings). If that is ok, then your ISP might be blocking port 25, so you will need to change that to 587 (SMTP port.) If that doesn’t help either, then you will need to use your ISP’s SMTP server and login to send mail.

If nothing works, then write me. Sorry for the inconvenience…