Diary of Pain:
My continuing DSL (and Fiber) saga

by Erik Olson

Goal: To have a high speed link in my home to which I can hook up a small Linux-based PC to act as a web/mail server and firewall to my home computer[s].
Spring 1998 US West announces availability in Seattle June 30. Having hosted some aquarium-oriented web pages through the graciousness of Caltech for the past four years, I see this as a possibility to finally relocate them to my house for $60 a month. I search madly through any and all documentation for evidence of static IP addresses, and find virtually nothing. I sign up my e-mail address on the Megabit web site to be informed when DSL becomes available.
June 1998 Discover comp.dcom.xdsl and the myriad of installation horror stories. Resolve to use a local ISP instead of uswest.net.
June, 1998 Active date pushed back to July 9th. I learn about USW's ISP list on the net, and check the site of every one for policies and prices. Most have nothing listed.
July 1st I call the friendly guys at Caltech who've been hosting my site, and get statistics. Holy crap, I've been getting 150,000 hits a week, or about 5gigabytes downloaded per month. On calling a few independent ISP's, they inform me there's no way in hell I'm going to get that for anywhere near $60 per month from them. I think they must have assumed I was setting up a porn site or something by the tone of voice I heard. Calling up USW, they say they'll rent blocks of 8 (really 5, once the router, netmask and network addresses are removed) IP addresses for $15 a month. Big guy 1, little guy zero.
July 9th Washington CO's go online today. 888-MEGA-USW is pretty busy the first couple times I call, but I get through after a bit. The first guy I talk to is glad to take my order, though my local CO doesn't come online until July 30th. They'll be able to come to my home August 14th. When I ask him about static IP addresses, it turns out that it's actually $30 per month, not $15. First I have to pay $15 per month to get "Internet Office", which then gets me the privelege of leasing 8 (really 5) IP addresses for another $15 per month. It also gets me a bunch of other crap, like extra mailboxes I won't be using anyway. But I say "sign me up!" and he transfers me to "Faith", who promptly forgets all that part about me wanting the IP addresses, until I tell her about it again. slowly. She says she'll make a note that I want Internet Office and the block of 8 (really 5) IP addresses, and sets me up for August 14th install, and says someone will call me before this time with all the information.
July 11th Gonna need some network cards. Happily, I got a few prehistoric ISA cards free in a recent company giveaway, which fit nicely in my prehistoric 486 "firewall" machine. I run out and buy a $32 PCI card for one of my other systems, and a surplus $25 mini-hub.
July 20th Stuff starts to arrive at my door. A medium-sized box contains the Netspeed-204 modem, a bunch of cables, a CD of software I'm never going to use, lots of little filters I'm never going to use, a wall-mount low-pass filter in a phone jack which I'm going to take apart and use as a splitter for the whole house, and a PCI network card which I'm now also going to use because I already bought a week before. In another letter, I get a piece of paper informing me of my e-mail account and that I need to set my "Windows or Macintosh" for DHCP. Uh oh. No sign of any obvious phrases such as "Internet Office", "Here's your 8 (really 5) IP addresses", "Registering Your Domain", etc.
July 30th A call into the tech support line reveals the DSLAM is active at the local Central Office, which means I should be able to plug in my modem and see it work. She says the Internet Office stuff should have arrived, but is not surprised that all my material seems to indicate I'm getting the other "normal" service. For kicks, I try connecting the modem. It doesn't do anything, except flash a few lights. Reading key bits of the documentation reveals that Netspeed ships the modem in Routing mode, but USW's service works in Bridging mode, so I have to connect a serial line to the thing, log in as root, and configure it with "set bridging rfc1483 enabled", followed by "write" and "reboot". Also interesting, is that I have to special-configure the flow control on the terminal software to "none". End-users are just going to LOVE this stuff. This must be what they send the guy out to your house for. Even with the modem configured correctly, I still don't get anything beyond a "link" light, so I'm guessing it's not hooked up to the ISP yet. I also learn that after five minutes, I can turn the modem over and fry an egg on its underside.
July 31 A week has passed in silence with no more material. Read on the net about USW losing orders from people, and am starting to panic. Calling the new DSL tech support line, I reach "Mellissa" who seems annoyed I'm calling two weeks before I'm even supposed to be installed. She tells me that I'll get the Internet Office stuff before the due date, and I will do all my ordering of static IP addresses off the uswest.net web site.
August 3rd Checking the uswest.net web site, I find no information on how to order static IP addresses, or for that matter, anything related to Internet Office at all. I'm starting to wonder if maybe they just ignore Internet Office orders, and expect you to call AFTER everything's in and you realize your order isn't right. Maybe they just want to be sure you REALLY REALLY want that part of the service. I resolve to wait for at least a week since the last call before pestering anybody.
August 6th Not a thing arrives, except a flyer in the mail telling me about this neat new fast service called ADSL that I can order from my phone company. I call up the tech line again, and get told to call the uswest.net line (888-777-9569), and ask for the "self-install kit" and also for the static IP address and Internet Office options. There I talk to "Brian", who tells me that they have me down for "Internet Access", not "Internet Office". He asks if this is something I ordered after the initial contact, and I assure him that no, this is now the third time I've asked for it, including the first call I made. Brian does some tapping on the keyboard, and assures me that it's being sent in right now to the business office, and I should get a confirmation within 24 hours.
August 10th Guess what? Not a peep. Today's journey begins first at uswest.net's line, where a pre-recorded voice tells me to call the business line (800-244-1111). Calling there (and waiting on hold for 5 minutes), I get transferred back to the uswest.net line quicker than I can say the word "Internet". The person there then tells me that the animal answering the other line is trained to recognize the word "Internet" and transfer everyone, and I have to avoid saying it, and instead simply shout "SALES, PLEASE!" but please, please, by all costs don't say the word "Internet" to the operator. This helpful person also informs me that once I'm connected to sales, I need to verify my Internet Office order is in the system by getting a "C-order number", and then call uswest.net back whereupon they will be happy to make everything all better. Fifteen minutes of hold music later, I get the main operator to transfer me to sales, and ten minutes more later, I finally talk to a sales person. The "C-order number" seems to act like a magic word, because suddenly they stop talking to me like I'm an idiot (similar to the change of attitude when I tell a telemarketer to put me on their "don't call" list). Calling back uswest.net, "Kellie" takes my C-order number and tells me to call back in two days for "Brian in Tier 2" to verify everything's set up. I still don't exactly know what I'm expecting to see.
August 11th This morning, I turn on the DSL modem again just for kicks. Ever since the 30th, it's registered nothing but a Link light. But today, testing with a borrowed laptop, it actually works... DHCP registers, and the packet sniffer tells me that Windows-98 is now happily spewing my private information out for the world to snoop. End-users are going to love this. In a fit of curiousity and boredom, I try out IP Masquerading on the firewall, which could actually let me run an entire company though that DHCP-assigned pipe should I want to. The connectivity does seem pretty fast, which is nice.
August 12th Calling the uswest.net line again, "Tyson" tells me "sorry, Dude, it still says you've got Internet Access," but waits with me on hold for 15 minutes for the big cheese to take care of everything. He's about to give up and fax the guy (and have me call back the next morning and complain again), when I remind him this is now my sixth iteration of doing just that. So we wait another five minutes on hold for Brian. Brian finally shows up and asks how long ago I asked for the upgrade in service. "July 9th", I tell him, "When I originally signed up." "Hmm, That is a while." I wonder if this is the same Brian as I talked to last week, or for that matter, if this is the "Brian in Tier 2". He is going to call "the people" and make sure it gets done, and will call me back (or have one of the night guys do it).
Later that day Message on my machine says I'm good to go.
August 13th I figure out that I have to go to uswest.net from MY computer in order to get their web page. When accessed from outside their network, one gets a redirect to a different server. Aha. I am somehow able to navagate through the maze of pages to "Customizing my account", where I "order" my block of 8 (really 5) IP addresses. I also sign up to have them "manage" my domain, before realizing I shouldn't have.

Around this time, I start noticing interesting things while running TCPDUMP on the ethernet line. Like that all ARPs are being bridged across the DSL line. Or that SAP packets are also being bridged, so I know there's some dork in Ballard running Netware on his Windows 95 machine. In fact, I know this every two seconds. Hope those other ISPs aren't charging their customer for these as part of their bandwidth cap.

August 14th My IP addresses are turned on, and the Internic shows the domain is active, but no message from USW about this. I'm so pleased with the Linux firewall code that I just leave it on, rather than using "real" IP addresses for my internal net. So the other 7 (really 4) IP addresses are wasted. Oh well.

A guy calls me at 2 in the afternoon to tell me that today's the day my ADSL service is due to be installed. After telling him I got it all working, he says "Great, then you'll never hear from us again!"

~August 17th Still no word from USW on the Domain, so I call to ask. The helpful (for real!) guy manually enters the stuff for me, and further confirms that their "support" is pretty limited: in general only one name per IP address mapping, no easy way to change things, etc. Some digging reveals I can change the DNS addresses to my own host (which is what I really wanted to begin with) just by notifying the Internic. The whole process takes about a day and a half (with all the confirmation e-mails required).
August 20 Finally, it all works the way I want. I have my IP address hooked into my domain's DNS. I'm getting mail. My web pages are active. Things are good.
September 10 While at work, I notice that I've spontaneously lost connection. Later in the day, I rip apart all my systems to try and figure out why. Did my ethernet cards break? Apparently not. Was it the gateway machine itself? No. Did I accidentally yank out some wires at home? Nuh uh. In fact, I even hook up my laptop directly to the DSL modem and plug the modem DIRECTLY into the external box where the phone line enters the house, and it's still a no-go. Going through two tech guys at US West Megabit, the second (and fairly intelligent) guy says my Netspeed-204 modem is probably defective & they'll send out a replacement Cisco 675 which should arrive by Tuesday. Gulp. After I get off the phone, just for kicks I stick a big honking fan on top of the DSL modem, and son of a gun if it doesn't start working again. Yipe! A major sigh of relief is had.
September 15 The Cisco 675 arrives in return-postage-paid box. It's the same physical unit as the 204, even the BIOS reports that its a 204. Guessing this includes the same heat generating potential as well, I take no chances and pop the fan on top right away.
November 4 The night before a weeklong vacation away from home, I lose connectivity again... darn WAN light blinking green. In between calls back & forth between the different numbers (the phone repair folks say they'll check the line; the hardware folks say bug the phone repair folks), I try hooking up the router directly to where the line comes in to the house (bypassing my wiring)... pulling the 5 volt plug out the back and re-inserting it causes the 675 to start wildly blinking its error light, and eventually, NOTHING! Just the green light. Oh boy. After call #4 (hardware guys), it's 10:30 at night, and I take apart the power supply to learn it's only supplying about 3 1/2 volts to the unit. Aha! The hardware guy had told me of many failures of this type... his exact words: "Why don't you try using a ZIP drive power supply?" Great. Some of us don't own ZIP drives, and even if we did, wouldn't the power supply be, say, powering the ZIP drive??? What I do have, though, are several surplus external SCSI enclosures with good ol' trusty bulky transformer-n-regulator supplies. A few minutes soldering it to some speaker wire and an appropriate plug, and the system's working again. Cautiously, I leave for vacation.
November 11 Returning refreshed and sunburned, amazingly, the system's still running! Calling US West for another RMA on the power supply, I first find out that they can't actually send me a power supply, but instead have to send the whole modem kit, and I will send it all right back, swapping out only the power supply. What a waste. I make the mistake of telling the guy I'm going to use my own heftier supply and keep the replacement as backup. He sees this as an annoyance... why do I need to do get it replaced at all if I'm not going to use it? I'm a little stunned at the lack of understanding of concepts like "defective merchandise" or "warranty". Meanwhile, I have constructed a little more permanent supply from a smaller (and not quite as exposed) switching power supply, and I get the side benefit of 12 volts to drive the cooling fan as well!
March 4, 1999 Things working pretty well the last few months, save the occasional US West server crashing on the other end, time provider Y2K problems, etc. Today a package was waiting at my door: another Cisco 675. Looks like they're going through and replacing everyone's old modems to the flashable 2.0.1 bios. I am a bit wary, because it seems just a bit "too good" with all the "nifty new upgrade, absolutely free" messages all over the box. And I am worried that with this revision of the BIOS they will finally limit the download speed to 256K (instead of the 640K that the old model actually did). Interestingly, the stats still say 640/272K, so they haven't updated the other side... yet.
August 16, 1999 I got two e-mails from people who can't reach my pages. They're both US West customers. Suspecting their DNS, I do a "dig @ns1.uswest.net thekrib.com" and find that they've still got the old entry in their table from the two days they hosted my domain, and nobody using uswest.net can actually look up anything in thekrib.com succesfully. Heck, www.thekrib.com redirects them to a US West server! Fifteen minutes on hold yeilds nothing, but they say they'll deal with it.
July 15th, 2000 For almost a year, no problems (or dealings) with US West, save the occasional service outage. But two weeks ago a piece of paper came in the mail warning me I needed to upgrade my DSL service to PPP mode instead of bridging mode. This is actually a pretty cool thing. For most people, they get Network Address Translation, which is exactly what I've been using on my gateway Linux box. NAT does firewalling, and lets you hook up as many computers as you want on the other side. Presumably this also means that I won't be getting that freaking IPX or NetBEUI brodcast traffic from all the Windows 95 Boxes on my street as well.

OK, so anyway, today I decide to deal with the upgrade. The special web page blows up on Netscape, so I do the familiar 888-777-9569 call, and get the same old "Press 1 if you have Windows 95 or 98, press 2 for Macintosh, press 3 for Windows NT". Last time I used the service, I hit zero because I wasn't using any of those. This time they've disabled zero as an option! It doesn't matter; the woman on the other end says "cool", and that she was thinking of learning Unix some weekend. She also explains that the upgrade web page downloads a little application to change the settings on the modem, and also notifies Qwest ("US West is now Qwest" messages abound everywhere since the buyout) to switch the other end of the DSL at 3 AM. Meanwhile, your service goes down until that time. She assures me this will work with the static IP addresses I have.

So in an effort to "be nice", I actually disconnect the modem from my unix firewall and hook it up to a Win98 machine, run through the myriad of recabling and update programs, and uneasily head off to bed at 11:00.

July 16th, 2000 I wake up from a dream about evil cats destroying the world to find the modem blinking and not at all working. Two hours, three tech support redirects, and two modem reprogrammings later, I learn why it still doesn't work, but also that they cannot switch it again until 3 AM tomorrow. Turns out the foolproof static IP address automation was not so foolproof as they thought. Though they switched me to PPP mode, they didn't hook up the IP addresses to the other end. Sigh. Hoping that this doesn't affect the start of my domain transfers.
April, 2001 Qwest announces that MSN will be taking over their ISP services soon. As there is perhaps 10 gigabytes flowing through my connection every month, I have a major lump in my throat over reprecussions of this tranfer.
July 19th, 2001 The DSL modem freaks out and detaches itself from the world, leaving the site offline for about 3 days. This is only the second time it's ever done this, and of course it's while I'm on vacation.
July 31, 2001 No MSN! Yay! Turns out that because I pay $30 more a month for their premium service and static addresses, they aren't going to sell me to MSN. At least not yet... I am overjoyed.
August 1-3, 2001 After a week of peace, I wake up to find the modem detached again. I reset it and wander into work; by the time I'm at work, it's toast again! Four more times my wife (normally with an 80-hour a week job but on a 2-month vacation) reboots the sucker. Strangely, a co-worker complains of the exact symptoms on the exact same dates! A quick Google/Deja search later, and I realize that the hype about the Code Red worm was not all coincidence with my DSL modem's fatalities: There is a bug in Cisco's firmware which can't handle the attack packet when directed to its internal web server. Hooray! I download a new version of the firmware (CBOS 2.4.1) and turn off the web-based configuration via "set web disabled" & "write". I think I'm saved...
August 4-6, 2001 ...but I'm not! Maybe it helped, but the modem's still crashing every few hours. All the tech support things on the Qwest site say to do what I did but nothing more. There is an 80-minute wait on their tech support line. I start poking around with other settings, such as "set web port 1234" and "set web remote 1.2.3.4". That seemed to do it. For the first time in a week, the darn thing's been connected for more than 24 hours.
July 26-28, 2004 I should have known... it all seemed too easy. A buddy of mine at work mentioned that DSL of ridiculous (no, ludicrous) speed is now available, and that I should consider more options. It has been bugging me that I only get 640/272 on this line. So I looked it up on the Qwest site, and sure enough, there's 1.5/896 available, and what's more, it's the same price as I'm already paying! Even better, they now have the option of ditching the useless qwest.net e-mail access that I'm paying $25/month for and only using to receive spam, replacing it with a $6/month "vanilla DSL" that lets me keep doing what I've always been doing.

So I call 'em up and find out that it's a two step procedure. Step one, which they performed flawlessly, was to disconnect the old service. Step two, which they managed to forget about, was to reconnect the new service. But somehow they manage to send me a modem before it all happens, so that I don't call inquiring about whether my order has been messed up. Instead, I am left completely unawares about this until the Internet suddenly shuts off Monday morning at 6 AM. "Okay!" I think, "now it will suddenly reconnect with my new modem!"

Hours pass. Nothing. No calls, no sign of change. Several phone calls later, I learn of the snafu. They assure me that they're "expediting" the order. "When will it be installed? I've got lots of servers and mailing lists and stuff that people are depending on! This afternoon?" The nonchalant tech merely says "don't bet on it." Tomorrow? "Don't bet on it." Great.

Interestingly, a side lesson learned in this all is not to upgrade too much at the same time. During these three days, an AMD 64 motherboard arrives, an eventual replacement for The Krib. I figure this is a golden opportunity to do some nasty upgrades while I have no connectivity. And nasty it is. Old databases no longer work with Fedora Core 2, hard drives that worked flawlessly now thrash. New mailing list software, that I'm sure will ultimately rock, currently sucks from lack of good documentation. It's late. I'm tired. I'm getting too old for this stuff!

November 2, 2004 Trouble with the phone line... noisy line, noisy DSL. Line tech supposed to come and fix it. This is definitely not my day. Have also been having a problem where the nifty new ActionTech modem locks up once a week, only resettable from home. Definitely not my month... or quarter.
December 15, 2004 Ever since July, when my DSL was upgraded to 1.5/896 and the new ActionTec GT701 modem, I have had to unplug and replug the blasted thing in every 4-7 days. More often than not, it happens in the morning right as I've left for work, and disconnects the sites, e-mail and other services completely for several hours at a time. I have been on the phone numerous times with Qwest tech support (thank God for my earlier notes on this page, I wouldn't even know what number to call first otherwise). A firmware upgrade doesn't help. Tech support person 1 tells me it's not just a firmware upgrade; I have to do a firmware "restore" and then do an upgrade. That doesn't work either, as four days later the modem is hung again. This morning, tech support person 2 then tells me it's because I'm running my own firewall that he can't ping the modem (ridiculous!), and I need to try unplugging the ethernet line from the modem to prove it (of course, he's not the same person I'll get when I call back later and prove it). I tell tech support person 2 that I'm ready to just buy an old Cisco 678 on EBay rather than deal with the ActionTec modem any longer.

After rushing home to unplug the ethernet line from the modem, tech support person 3 thinks that because they've had problems with Macs and long filenames, and Macs are based on Unix now, and Unix is kinda like Linux, that the problem is because I have Linux (as he explains this, I am thinking in my head..."If she weighs more than a duck, then it means she's a witch!"). He also offers the alternate theory that the problem stems from me running the ethernet interface into the modem at its full 100 mbps (apparently an earlier ActionTec model choked at 100mbps bursts). Finally tech support guy 3 puts me on hold and 10 minutes later comes back on and announces that his buddies in the pool are saying that the ActionTec modems suck for running reliable servers, and that I should go find an old Cisco 678 on EBay.

December 17, 2004 Got an old Cisco 678 on EBay.

It works great, just like my 675 has been working great for the last 5 years (OK, except for that Code Red thing in 2001). Got it cheap, practiced the upgrade on an identical unit at work (which I broke for a few hours...yay, more time with Qwest tech support!), got it right the first time with my setup. If you don't see any entries for a while, that'll be a good thing...

2014 While waiting for fiber to come to our neighborhood, I repeatedly get things in the mail telling me that I can get amazing speeds up to 5mbps. But upon calling the Centurylink SalesDroid network, they get confounded by my use of static IP addresses and tell me that it is impossible to upgrade from my 1.5mbps/896kbps arrangement that I've had for ten years. I get on the e-mail list to be notified the second fiber comes to our neighborhood.
July 2015 Service randomly cuts out at 7 AM. Calls to tech support are met with "There's nothing wrong indicated in your area." And then "But other people are also calling, so maybe there is." They agree to send a guy out to look at the phone. Two days later, a guy in a van does some tests on the line, determines that yes, it's not working at my house. He also decides he needs to call for a ladder truck to check things out at the pole 20 feet away. I ask why they don't just check at the central office, and he tells me that if they thought there was something wrong at the central office, they would have checked. So away he goes. Later that day at work, I get on Centurylink chat, and the guy there actually checks with the central office and finds that they didn't re-enable the DSL port that goes to my house. He re-enables it, but indicates the modem at my home isn't responding. I figure that the guy in the van didn't reconnect my phone when he left, and sure enough, the whole box is still open and disconnected. So I fix it myself, and am good to go. The chat guy also hooks me up with 5 MBPS service, which the previous two sales calls insisted was "impossible".
October 2015 Some sales contractor guys canvassing our neighborhood announce that fiber is now available. They tell me it's been active for about a month. (Thanks, Centurylink notification e-mail that never came!) They sign me up, but are once again tripped up by the static IP addresses. I am ultimately assured that everything will go smoothly and my static addresses will transfer over. I'm expecting three days downtime. Happens November 21st.
November 2015 - January 2016 Fiber installed.
  • Takes two days and three different techs. Service out almost 24 hours.
  • Get re-charged for hooking up my static IPs to the new service. Three calls to get the refund.
  • "free" modem turns out to be a $10/month rental. Get to learn how to use Tomato and a special VLAN to replace it.
  • Exciting change-over to single static IP from the 8 (really 5). But costing $10 less per month.
  • Centurylink's phone tree is as bad as it ever was. On hold for two hours for one outage to be told they know about it and there's nothing I can do until it's fixed.
August 2017 IPV6!
July 2019 1Gbps Upgrade.
  • CenturyLink requires me to remove my static IPs before performing the upgrade, throwing the domains into darkness for almost three days, and I'm sure adding an unnecessary $75 installation fee to my next bill.
  • They send another "modem" that I specifically asked NOT to have. Back it goes!
  • When I add a static IP back in afterwards, reverse DNS entry does not propagate...for like 5 days. Meanwhile, can't send e-mail to about 30% of domains, which reject or defer anything without a reverse PTR. Four calls and chat sessions:
    1. Wednesday evening, after 45 minutes: "Umm, it's not letting me change it. Oh nevermind, they said it takes 72 hours to process your request."
    2. Thursday morning, almost immediately: "Dude, wait 72 hours." Me: "But that puts me in the middle of the weekend, and you guys will be closed!"
    3. Sunday morning: "I don't see you having a static address. I checked many times. But while I've got you on chat, how about we talk about upgrading your service and add DirecTV" Me: "WTF?"
    4. Monday morning: A very nice woman admits that she doesn't get these sorts of issues on the residential side and she's really just winging it. The most refreshingly honest interaction this entire time. Unfortunately, they can't fix it either. Sets up a ticket to be responded to within 24-48 hours, and I go a little ballistic. She attempts to escalate it.
    5. Monday, 1/2 hour later: I get a text on the road that the DNS entry is propagating.
  • Learning about cut-through forwarding and its incompatibility with hairpin routing. Still love using Tomato as my router. But without cut-through enabled, the AC3200 only forwards at ~300 Mbps. Solution is to not use my internal host for DNS queries, but use the router itself, which I kinda was already doing. The router runs dnsmasq, which can be embellished with local host names that resolve to internal addresses. Cool.