Yesterday, we had some severe Internet access issues. As a matter of fact, we didn't really have Internet access, unless you consider this:
Internet access; which none of us here did at the time. Annie called it dial-up speed, and she was dead-on the money.
I
had been noticing increasing problems during the week, but Sunday kind-of capped things off, forcing me to do something I dread doing...
calling Verizon OnLineTech Support.
If you don't feel a shiver reflecting up and down your spine, like a standing wave, there's something seriously wrong.
Remember that I said that I had been noticing problems last week?
Yeah, well, over the week, I did what competent Net Admins would do...I spent time trying to diagnose and isolate the problem. This put me ahead of the Verizon Tech Support flow chart, of course, and effectively isolated the issue to someplace between the DSLAM and the Verizon Primary Name Server. Not much isolation, I'll admit, but the key issue is that the problem was on the public/Verizon side of our assigned DSLAM. Not a problem we could control at all.
I ran a quick series of ping tests of the network interfaces, then ran a long series of traceroutes and pings at 4.2.2.1. I was getting 90% duplicate packet returns from hop 4. In other words, for every 1000 packets I sent out, I received 10,000 packets back, 9000 of those being packet sequence duplications by the #3 router after my public interface.
I also ran the above SpeakEasy test, obtaining the results--after far too long a wait.
Off to phone Verizon...Oh Boy!
After about 45 minutes on hold, I got a hold of a CSR who began with the,
"OK, sir; I'm going to ask you to turn off the power to the DSL Modem (It's really an ATU-R),
and turn off the power to your router and switches and all of the computers" routine. This was, of course after I explained to his understanding the scope and structure of our network.
{You could have just said, 'Power cycle the network' and it would have been less of a waste of my time to listen to.} "Nope. I'm not going to do that, because I already have today, as well as having done so about 3 times a day for the past week. I've also done the 'release and renew' number several times this week too. I'm trying to report a problem with one of your routers."
"What Operating System are you using?"
{Are you freaking brain dead?} No. I did NOT say that out loud.
"It doesn't matter. Just ask me for the information you want. I know enough to get it for you."
"I have to know what Operating System you're using, sir."
"No you don't. You're simply looking for an excuse to drop the call."
"Sir..."
"Fine. Linux."
"We don't support Linux."
{OH! Gasp! Gasp! What the Hell did I just SAY? Going postal was beginning to look good, just then.}
"Unless you can connect a Mac OS or Windoze OS computer directly to the Modem, I will have to terminate this call, sir."
"Give me Level 3 tech support, or a supervisor, please. You have a problem with a router 4 hops from me."
"I'm sorry, sir. I can't do that."
"Let me speak to Mike in Level 3 support, please."
{Line went dead.} {Dial tone.}
{1.877.483.9322}
{Same song, second verse, with Sylvia, except she did slip-up and reveal that the last CSR deleted all the notes on the immediately preceding call.}
"...connect a Mac OS or Windoze OS computer directly to the Modem."
{What a pain in the butt!}
"...you'll have to call back when you get a windoze machine, sir."
That'll be a cold day in Hell.
I got the Linux-based laptop out and took it over to the ATU-R cabinet, 8 feet from the entrance bridge/NIB/SNI (call it what you will), and connected it straight to the Westel 6100 with a bit-o CAT-6.
I did a simple ping of 4.2.2.1 and got a consistent 90% duplicate packet reception situation, and just let it run while I ty-wrapped a bunch of (disconnected) wires together in another cabinet before calling the people who fan themselves with workgroup process flowchart notebooks. I intended to just tell them that it was a windoze machine and give them the information that they needed, and do what I needed to do to duplicate what they would want a windoze user to do to accomplish a specific goal.
As I was working, I glanced over at the console session I had running, and noticed that I was coming up with 100% packet losses. I was working on wires that had nothing to do with what was active for broad band access.
I got off the ladder and walked over to the laptop, on the other side of the room. Yup. Looks like I'm talking to nothing at the end of the wire. I Ctrl-C'ed out of the command and took the interface down then back up again. When the interface came back up, I had a radically different IP address than the one we had been using for the past few months.
"Oh! Ah-Ha!" Somebody listened to me enough to know that they screwed-up, but wouldn't cop to it on the phone.
Mind you, the ATU-R was never power-cycled. It had to have been forced into a release and renew from the Verizon side; which means that somebody was working on either the copper or the DSLAM. I re-ran pings and traceroutes, and discovered that the duplicate packet issue was completely gone.
I jumped over to SpeakEasy.net and re-ran their speed test routine...heh...
Problem solved.
I went back to making my wiring cabinets look the way they should have the day I was in too much of a hurry to keep it neat; then I put my network back the way it was before calling Verizon in the first place.
Funny. It wasn't necessary to follow their little troubleshooting compulsions. It was only necessary to become near-offensive, and to light a fire underneath their posteriors by telling them exactly where the problem was--in their network.
To Mike, in Level 3: "Dude! Hope you're still having fun with Slack. Email me your direct line, please. These front-line CSRs are worthless!"
