Hopeless Geek

I signed on with JVDS quite a while back, perhaps a year ago. It was right around the time I moved this site to Drupal. JVDS’ server, the one I chose, runs FreeBSD and uses simple jails for their virtualization technology. For easy-going people this is good, and in all honesty it worked fine for a long time. Now and again some fuckhead would get a broken program stuck and take over the machine, but that’s what you live with. It never ran out of memory, and when it had 16 people on it things were excellent.

Then they got bought ever-so-shortly after I joined. In time I noticed more and more disks being added (jails) and, today, that number sits at 38 jails on the one machine. Load is always at 3.00 or higher and disk access times are insane. So, CP, MG, OBS, AA, and some other sites all ran very, very slow.

It took some shopping around, but I decided the best way to defend against overselling was to pick a place that used a virtualization technology that could not be oversold. It appeared that Xen was the answer to that. Specific CPU time, RAM amount, and disk amount for each VM, as well as SSH console access and so on. Looked great, so I looked for providers.

In the end I narrowed things down to unixshell# and Quantact for the providers, based on price, offerings, and searches of “[name] sucks” in Google. I stared at the price lists for a few days, in-between projects and such, until deciding to try Unixshell first.

I requested an account and 22 hours later I had one. Warning number one.

After getting said account I started moving services to it, starting with mail for CP and MG. That worked very well, and the server appeared very responsive, so I moved MySQL over to it and had Metis (the JVDS machine) use Triton (the Unixshell machine) for the database. The speed improved! Pages on CP and MG were loading very fast compared to how they had been, and load on Metis went down (showing I was one of the primary users… ). So, things were good.

Then I moved DNS, Apache, and Jabber over to Triton. Instantly there were some small problems. It ran out of memory a lot, but that was my fault as I’d been on a jail before and tuned for speed rather than memory. Some tweaks to my.cnf and Apache’s prefork stanza and we were in under 128MB for the whole schebang and page load times were reasonable. Not much faster than Metis, but reasonable.

A week later things come to an absolute fucking stop. I reboot the VM from the control panel and investigate. Things were running out of memory! “I just tuned this bitch and it was working,” I thought to myself. So I ran some tests using sar and discovered there was a huge iowait issue. Everything that used the disk caused 50-75% iowait. MySQL made it hit 100%. Using “sar -W” I determined this wasn’t a swapping at first, only as things ran. This was the disk. Here’s how that breaks things:

1. HTTP request comes in and Apache spawns a process (1 process)
2. Drupal executes some number of SQL queries, possibly causing MySQL to spawn a process (2 processes)
3. The system is slow responding to the disk access requests, causing those processes to tie up memory longer than expected as another HTTP request comes in, spawning an Apache process (3 processes)
4. Drupal runs some queries, MySQL spawns (4 processes)
5. Increased memory usage causes some swapping, which is disk access, which slows the queries even more and they start stacking (5-10 processes)
6. More is swapped for these new processes, disk is slower, VM is slower, MySQL is slower…
7. Out of memory, out of swap.
8. Death.

Great, eh? Check this out:

00:21:01	CPU	%user	%nice	%system	%iowait	%idle
00:21:02	all	1.98	0.00	1.98	96.04	0.00
00:21:03	all	0.99	0.00	0.99	98.02	0.00
00:21:04	all	0.99	0.00	1.98	97.03	0.00
00:21:05	all	1.98	0.00	0.99	97.03	0.00
00:21:06	all	1.98	0.00	1.98	96.04	0.00
00:21:07	all	1.98	0.00	3.96	94.06	0.00
00:21:08	all	0.99	0.00	7.92	91.09	0.00
00:21:09	all	1.98	0.00	6.93	91.09	0.00
00:21:10	all	1.98	0.00	0.99	97.03	0.00
00:21:11	all	0.00	0.00	0.99	99.01	0.00
Average:	all	1.49	0.00	2.87	95.64	0.00

So, I send in a ticket. A very detailed ticket. I check the “emergency” checkbox.

I wait 4 hours.

I wait 12 hours.

I wait 24 hours.

I send a very nasty note. At this point I don’t even have an auto-reply. I don’t have a “we’re looking into it.” I have squat. I know they have it because it’s in the ticket system on the web.

So, I wait 12 more hours.

I send another nasty note.

I get a nasty note.

Your issue isn’t an instant-fix at the moment. I/O wait problems have to be looked at carefully by Engineering before we can respond with any results. Otherwise, all we can do at the moment is tell you we’re working on it at the moment.
—
-davidk
Tektonic Network Solutions | support@tektonic.net
Unixshell# | support@unixshell.com

Oh, you little asshole.

Yes, but you haven’t even done that. I’ve received absolutely no communication what-so-ever from your company since I filed those tickets. I don’t know that someone’s read it, or working on it, or even gives a damn that it’s a problem. Yours is the first communication I’ve received that even reiterates the complaint!

I might wind up leaving on principal alone at this point; this is horrible. I’ve been down for three days with not even an acknowledgement of my complaint. That’s SHITTY service. Absolute bottom-of-the-barrel. There’s “unmanaged” services and “unsupported” services. I feel like I have the latter with this kind of treatment.

So … you’re saying someone knows? Someone cares? Do you have any information at all? Fixes are one thing, and can take time, but letting the customer know you haven’t completely forgotten them is a whole other thing, and something you folks seriously need to work on.

I felt better. Well, until eight minutes later:

Not trying to be amusing here or anything, but our general policy is to try and work on an issue first then respond after that. We tend to get things done faster this way, as we spend more time trying to fix an issue, and gain more insight into an issue. You may not agree, but this policy has been shaped over several years of our business in this field. If this method of resolving an issue is not acceptable there are always other routes that you may take.

Oh? Really? It’s faster to get to work on it rather than send an email saying “Oh, crap. Yeah, let me look at that…” and then go work on it? Realize something: it has been three days at this point.

So, I signed up with Quantact. I was using the account three hours later. A day later, I had it all working and had all the services moved over and it’s been great since then. All my sites are there now and running fine. Average load is under 0.20 and I’ve hit it with ApacheBench several times now. It takes the hit and then it’s back to normal, as one would expect.

So I deleted things of interest off Triton, said hello to Oberon (Quantact’s VM) and send the following back to Unixshell:

On Dec 14, 2005, at 10:50 AM, sales@unixshell.com wrote:

Not trying to be amusing here or anything, but our general policy is to try and work on an issue first then respond after that. We tend to get things done faster this way, as we spend more time trying to fix an issue, and gain more insight into an issue. You may not agree, but this policy has been shaped over several years of our business in this field. If this method of resolving an issue is not acceptable there are always other routes that you may take.

I was unaware that it took several hours out of your technician’s time to reply back with a courtesy “we’re looking into the problem and will notify you when we’re done.” Indeed, as far all of my tickets are still very open with no new status at all. It must take an incredible amount of effort to enter in a five-to-ten word courtesy response.

As such, I offer my sincere apologies for having that minimal expectation of customer service. I’ll let you and your admins get back to figuring out which end of the computer the power goes into, because God knows it’ll take several hours to handle that and it’s probably taken an hour or two out of your work day to just read this message. I’ll not interrupt you any further with requests for status on restoring the service I paid for and never quite received.

I’ve spent the last week working to remove my information from my VM and am complete. Consider this a notification that I want the account closed. I would have told you as I started to work on it, but it would appear you and your company don’t agree with giving people a chance to deal with a situation ahead of time, so here we are.

Merry fucking Christmas, ass nuggets.

And that ends my fiasco and explains the several days of downtime we’ve experienced.

Moral: If you ever want to get a VPS, do not use Unixshell or their parent Tektronix. If it works, it works. If it doesn’t, God help you.

I’m also not alone in this; their message boards (something else to check when you move somewhere!) have several people questioning the turn-around-time on a ticket they’ve put in and it has several people complaining of iowait issues on other VM hosts than the one I was on. They’re not ready for the traffic they’re seeing and should stop selling machines and deal with what they have, but just keep on trucking instead. That’s bad management, and that’s what will kill them.

Update: Lovely.

Your account has been terminated. And for future reference, it does take several hours, because you are not our only customer, out of our several THOUSAND customers, we have quite a workload. Just an FYI for moving to your next hosting provider, sometimes the best way to get someone to help you isn’t to call names and belittle people, politeness goes a long way with our engineers, the same as anywhere else.
—
-Ryan M. Adzima
Unixshell# | sales@unixshell.com

Even afterwards, they manage to be assholes. So, fine, I can as well:

I agree that it can take several hours to complete a task. I do not agree that it should take several days to provide first-response to a customer on a technical issue that has been classified as “emergency/site down” by the customer.

Perhaps if you have thousands of customers you might be better off a few more people to properly support them. Just an FYI for the next customer that leaves you because your customer service is absolutely abhorrent.

What makes this all the more silly is that I work in an enterprise setting, and I know what customer service should be. “Server down” requests should be given a basic response within one business hour. Four hours is on the long end, but also acceptable. It took your “business” over 72 hours to respond, and then did so snidely.

You folks have a lot to learn about professionalism and customer care. You sell shit, you treat people like shit, and every interaction I’ve had with you has reeked of it.

And from here on out, procmail will dump anything from unixshell.com. I highly suggest people avoid this company for the next year or two until they get their act together … if they ever do.

Average: Select ratingCancel ratingPoorOkayGoodGreatAwesome

Average: 5 (1 vote)