Outage Today - Caused By Network Problems

[Admin]

Ladies and Gentlemen, dear fellow pen nutters,

 

Quite a few datacentres were plagued today with outages due to problems with backbone connections to the internet, or rather, the equipment connecting the datacentres with the backbone. This outage started around 11:10 UTC / 07:10 EDT this morning.

 

Unfortunately FPN appeared to be hosted in one of the data centres which were affected. It also appears that things are fixed now, but we have been told that there still may be some intermittent problems.

 

Obviously there is very little we can do about this, but bear it and grin . At least we still have our fountain pens and notebooks to go back to, unlike many people who were looking for their favourite websites in vain - literally thousands of websites were down.

 

Anyway, we're back, and let's just enjoy again!

 

Warm regards,

 

The FPN Admin and Moderator Team

 

 

[rwilsonedn]

Once again, thanks for staying on top of all this for us, you guys!

It is appreciated, probably more than you realize.

ron

[esterbex]

Once again, thanks for staying on top of all this for us, you guys!

It is appreciated, probably more than you realize.

ron

+1K!!!

[wimg]

It's a bit of a pain though if all you can do is occasionally chat with an engineer who is waiting for the engineers of the peer network company to come up with a solution, while frantically trying to sort this out ...

 

I don't want to think about how many thousands of websites were down today because of this problem, which affected quite a few data centres. Even a famous chat service was down for me for some time...

 

Warm regards, Wim

[inkstainedruth]

Wow. Good that we're back up and running. I've been away from here for the better part of a week. I seem to miss all the excitement....

I hadn't realized there was a problem because by the time I logged on today (late morning) the sites I needed to be looking at were okay; you know -- "important" ones: isellpens, Goulet Pens, Ebay....

Ruth Morrisson aka inkstainedruth

[notbob]

I'm no sysadmin and I don't pretend to tell you folks your business, but FPN definitely needs to start looking for another hosting service. Seriously.

 

BTW, this message editor is pretty messed up. It took me 3 tries to change the font of this msg to Georgia font and then when I finally looked at the msg preview, the editing function crashed. Things are far from back to normal.

[lapis]

Thanks from here as well. Personally, I don't think it matters so much who is responsible for the outage, it matters so much that our admins have the knowledge necessary, and can and will spend their time and energy to fix things!

 

Fritz

[OakIris]

I wondered what happened; none of the other sites I frequent had a problem yesterday so I just figured that FPN was the "victim" once again. This time though, the site seemed to have totally disappeared, so that was worrying!

 

Glad to see the site is back up, but I must say that, thus far, 2013 has not been good for FPN.

 

Holly

[jacksterp]

Happy for all things are back to "normal."

[ethernautrix]

Heh. Didn't notice. Every time I came over here, here was here. :-)

[OakIris]

Heh. Didn't notice. Every time I came over here, here was here. :-)

lol - well then, you were lucky! Every time I tried to get here yesterday, here wasn't here, here didn't exist! (Luckily, I was able to go there instead....)

 

(I always enjoy seeing what new avatar you come up with, Lisa - your current one makes me chuckle!)

 

Holly

Edited May 30, 2013 by OakIris

[RLTodd]

Oh, gosh, this is just short of awful. ( I have to keep perspective because there are always things happening in the world that are really awful, like wars & plagues & floods & such.)

 

I keep thinking of that old thing, "For want of a nail...." Or, House of Cards, or Jumanji for younger folks.

 

I wonder if our systems, some of which we depend on to keep living, have become so complex that one pin pull out somewhere along the line collapses the whole thing?

 

I am not a believer in coincidence.

 

I am not in panic mode, but I am beginning to get a bit worried.

[wimg]

Well, we got an official apology today.

 

However, it appears their cloud systems were ok, as they were on another part of the infrastructure, with their own internet connection, and had priority access to the internet backbone as well - they should not have mentioned that in their email, as that is really upsetting.

 

Considering what has been going on for the last three months, since they migrated to a new data center, which, BTW, is still being built up, we already had been looking into alternatives.The problem is that a move to another host provider will mean down time, likely for a few days, considering the amount of stuff to be migrated across. A full backup currently takes 5 whole days to download (!), and an upload probably 10X as long (ouch), so we would need to find a solution for that as well.

 

The team they have which is looking after us is absolutely fantastic, but if they don't have the tools to help us, it is going to stop somewhere. Also, all of these problems have cut a HUGE dent in our budget, as we had to get in a few (paid) consultants, and also had to temporarily get another server.

 

So, I guess we will be moving, once we find a solution to migrating all data across in a reasonably fast fashion. We've got some ideas as to the approach to take - need to work this out further, though.

 

Warm regards, Wim

 

 

[wimg]

Oh, gosh, this is just short of awful. ( I have to keep perspective because there are always things happening in the world that are really awful, like wars & plagues & floods & such.)

 

I keep thinking of that old thing, "For want of a nail...." Or, House of Cards, or Jumanji for younger folks.

 

I wonder if our systems, some of which we depend on to keep living, have become so complex that one pin pull out somewhere along the line collapses the whole thing?

 

I am not a believer in coincidence.

 

I am not in panic mode, but I am beginning to get a bit worried.

It appears something somehow went wrong with the ip configuration tables in some very highly specialized routers, and not just one, but a whole bunch of them as used by the peer network provider, one which is one of the top 5 biggest peer network providers worldwide, so yes, I am getting worried too.

 

For those not in the know: this is about access to the (internet) backbone, which essentially is the internet super highway. Now, all internet traffic congregates from, so to speak, internet dirt roads, provincial roads, state roads, villages, towns, cities, mega-cities and metropolises right onto the internet highways, before being moved along to the internet super highways. What went wrong here is that a whole bunch of equipment controlling access from the internet highways onto the super highways decided to block all traffic, instantaneously and simultaneously, yesterday morning around 7 am EDT. And this could only be fixed about 12 hours later.

 

Yeah, worrying indeed.

 

Warm regards, Wim