WTF ... IS WTF!?
We are a collective of people who believe in freedom of speech, the rights of individuals, and free pancakes! We share our lives, struggles, frustrations, successes, joys, and prescribe to our own special brand of humor and insanity. If you are looking for a great place to hang out, make new friends, find new nemeses, and just be yourself, WTF.com is your new home.

Networking Life sucks thread.

MaxPower

You're my number two
Staff
16,851
3,350
487
#1
So today I took off from work (my day job) to conduct some business for my private consulting practice. I was out in Piscataway, and Edison, working with a Telecom provider that shall remain nameless, some of you may know who I'm talking about. I'm having lunch with some of the PNE team when my cell phone rang. The voice on the other end said, "Our phones are down". Now, due to downsizing, I am the only one qualified to handle such emergencies. So in the middle of lunch with the customer, I had to excuse myself and turn over the proceedings to my partner.

After talking them through some PBX and Channel-bank troubleshooting, I determined that the problem resided at the carrier, or at least beyond our CPE. Following this was a barrage of phone calls to AT&T (our service provider) and Cablevision Lightpath (our Local loop provider). All parties involved, swore up and down that the Circuits tested good to the Demarc.

I finished my business and drove the 2 hours back to my office.

Here's where personal experience comes in. NEVER TRUST THE SERVICE PROVIDERS.
Ok, I did some reverse testing out to the MUX. No tone and couldn't even see the MUX. So I contacted building management and asked them to provide me with the Suite that contains the common demarc. (We occupy a Lit building, which means we have fiber right off the street and the multiplexing and copper conversion gets done here. An OC-3) they told me it was "Company XXXX". So I walked over to Company XXXX, to find they had moved out. The place was vacant. Like thieves in the night, they left. I let myself into their former suite, and walked back to their former Network operations center, to find it had been ransacked. All their equipment was understandably gone, and All the Demarc equipment had been unplugged including the FD-6W, which is responsible for the fiber OC-3 breakout to copper T-1's. Walking back to my office I retrieved some extension cords and power strips. I plugged all the equipment back in and can swear I hear the sound of 6 different companies in the complex rejoicing in unison.


So our Voice T came back plus the data T’s of our neighboring companies. What have I learned?
1. My current day job needs to pay me more. They had no phones for 4 hours, and I resolved it after being back onsite for 15 minutes. They would obviously crumble to a useless mass of non-communicato’ in a matter of hours without me.

2. The service providers don’t include their Multiplexers outside the CPE when testing circuits, or failed to in this case.

3. Get building management to install a lockable NEMA enclosure around the Shared Demarc.

4. I need to be paid more.


When I went for my MCSE, there was a Microsoft propaganda poster on the wall at the school that I never forgot, and today I know why. It’s times like these (where everyone is too stupid to plug something in) that makes me reflect on those days.

Microsoft Propaganda poster ~”How would you like to save the day…..Every day?”


~ Fuck the fucking fuckers.



.<-----
 

Jung

???
Premium
13,980
1,398
487
#2
Goddamn those fucking demarcs. :(



Last real problem I had was a client's server that was compromised. This client had chosen to outsource his hosting, but have us manage it. (don't ask why :rolleyes: ) I got to work to find an inbox full of frantic emails about insane network usage and such.(for a mail cluster) Come to find out, this guy's box was rooted via an SSH2 exploit, and was scanning through a 5MB text file of US net blocks looking for the same SSH exploit. Not only that, but had roused a sizable amount of zombies that were reporting to the server in question. Turns out, it was trying to form/complete a botnet for some kind or irc DDoS attack.

Well that was fun! The colo unplugged their server as soon as they noticed the activity. I got to call their colo (The Planet, TX) and have them ship us the drive. Rebuild a new drive from backups (our onsite backups weren't up to date) and cross ship it to the colo, then mount the compromised drive on a dev box, salvage what data I could and rebuild the drive from back ups. Luckily the colo had onsite backups to cover what me didn't have.

Come to find out that a good bit of GNU utilities (ls, cat, tail awk, grep, et al) had been compromised as well. That was a cluster fuck of a few days right there.

Btw, this wasn't a server I normally managed either. Yay for shitty admins who are on vacation when their babies blow up. :rolleyes:


Bonus pic of insane mail server bandwidth.
 

Broken

Member smoked too much weed!
3,891
0
0
#3
Ladies and Gentlemen, WTF presents for your viewing pleasure Descents adoptive "" Parents ""


$ = teh ./ of <> evil.

Am3|\|
 

Descent

Hella Constipated
7,686
109
157
#4
I kinda made a few words out there. Now I'm gonna go look up the rest.

By 12th grade I'll have a worthless certification - YAY! Just what I always wanted!

And I doubt I'll be able to perform all that Max, but I am going to learn Cisco routers in a room with half a million worth of equipment.
 

LiberatioN

Trance Addict
1,432
0
100
#5
ewww Cisco Networking...I took a class in HS. It was boring; they basically teach you generic stuff you could learn on the internet (namely, this forum :lol: ) Sad thing is, I probably comprehended about as much as you did :(

Subjects covered in Cisco Networking classes:
-Basic DNS structures
-IP Configuration
-Setting up LANs with access to WAN
-Configuring switches/routers
-Hardware Identification
-Business Solutions

If you want a couple years of lectures I had via powerpoint, PM me.
 

Descent

Hella Constipated
7,686
109
157
#6
LiberatioN said:
ewww Cisco Networking...I took a class in HS. It was boring; they basically teach you generic stuff you could learn on the internet (namely, this forum :lol: ) Sad thing is, I probably comprehended about as much as you did :(

Subjects covered in Cisco Networking classes:
-Basic DNS structures
-IP Configuration
-Setting up LANs with access to WAN
-Configuring switches/routers
-Hardware Identification
-Business Solutions

If you want a couple years of lectures I had via powerpoint, PM me.
Community college, buddy.
 

Jung

???
Premium
13,980
1,398
487
#7
LiberatioN said:
Subjects covered in Cisco Networking classes:
-Basic DNS structures
-IP Configuration
-Setting up LANs with access to WAN
-Configuring switches/routers
-Hardware Identification
-Business Solutions
You forgot the two most important parts of networking: OSI model and subnetting.
 

MaxPower

You're my number two
Staff
16,851
3,350
487
#8
Don't forget password recovery ( Configuration register 0x2142 ) :)
 

Jung

???
Premium
13,980
1,398
487
#9
MaxPower said:
Don't forget password recovery ( Configuration register 0x2142 ) :)
Didn't they fix that in IOS 11.something?
 

MaxPower

You're my number two
Staff
16,851
3,350
487
#10
update: Events dated 08/02/2005

So, as it turns out we narrowly a missed having the same events occur as we did the other day. Yesterday the President of the company that moved out of the suite where our OC-3 demarc is located, came to me with disturbing news. "Max, LIPA (Long island power authority) is coming to shut off the power to our suite;" he said. Which would have left us with no power to the FD-6W OC-3 demarc, or the MUX. We would therefore be in the same situation as the other day, where our Voice T-1 would be down and the company leasing space from us would have no Voice or Data.

After a barrage of phone calls I somehow managed to talk the Landlord into absorbing a $2000.00 /Month Electric bill, until the matter can be settled. The ball is in his court now. The options are having the equipment moved, or putting them in a secure enclosure and having the Building Management pay the electric bill until the suite can be leased to new tenants.

For anyone looking to break into the IT field, this is a good lesson. Your job (depending on your position) may end up being 90% Technical, and 10% Politics.