Jump to content


Photo

Outage Today - Caused By Network Problems


  • Please log in to reply
13 replies to this topic

#1 Admin

Admin

    The FPN Admin Team

  • Admin

  • PipPipPipPipPipPipPip
  • 475 posts

Posted 29 May 2013 - 23:48

Ladies and Gentlemen, dear fellow pen nutters,

 

Quite a few datacentres were plagued today with outages due to problems with backbone connections to the internet, or rather, the equipment connecting the datacentres with the backbone. This outage started around 11:10 UTC / 07:10 EDT this morning.

 

Unfortunately FPN appeared to be hosted in one of the data centres which were affected. It also appears that things are fixed now, but we have been told that there still may be some intermittent problems.

 

Obviously there is very little we can do about this, but bear it and grin :D. At least we still have our fountain pens and notebooks to go back to, unlike many people who were looking for their favourite websites in vain - literally thousands of websites were down.

 

Anyway, we're back, and let's just enjoy again!

 

Warm regards,

 

The FPN Admin and Moderator Team

 

 


This account is unmanaged.
Please direct questions and comments to FPN Admin email, or directly to admin Wim (wimg).
 
Thank you very much in advance.
 
Warm regards,
The FPN Admin Team

#2 rwilsonedn

rwilsonedn

    Antique

  • Member - Gold

  • PipPipPipPipPipPipPipPipPip
  • 2,707 posts

Posted 29 May 2013 - 23:57

Once again, thanks for staying on top of all this for us, you guys!

It is appreciated, probably more than you realize.

ron



#3 esterbex

esterbex

    Rightly seasoned

  • Premium - Emerald

  • PipPipPipPipPipPipPipPipPip
  • 2,787 posts
  • Location:Rock City Arkansas
  • Flag:

Posted 30 May 2013 - 00:06

Once again, thanks for staying on top of all this for us, you guys!

It is appreciated, probably more than you realize.

ron

+1K!!!


God is seldom early, never late, and always on time. ~~Larry Brown

#4 wimg

wimg

    Stip Etruria nut :)

  • Admin

  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 18,736 posts
  • Location:Maastricht, Netherlands, EU
  • Flag:

Posted 30 May 2013 - 00:10

It's a bit of a pain though if all you can do is occasionally chat with an engineer who is waiting for the engineers of the peer network company to come up with a solution, while frantically trying to sort this out ...

 

I don't want to think about how many thousands of websites were down today because of this problem, which affected quite a few data centres. Even a famous chat service was down for me for some time...   :)

 

Warm regards, Wim


the Mad Dutchman
laugh a little, love a little, live a lot; laugh a lot, love a lot, live forever


#5 inkstainedruth

inkstainedruth

    Donor Pen

  • Member - Gold

  • PipPipPipPipPipPipPipPipPipPip
  • 4,576 posts
  • Flag:

Posted 30 May 2013 - 01:09

Wow.  Good that we're back up and running.  I've been away from here for the better part of a week.  I seem to miss all the excitement....

I hadn't realized there was a problem because by the time I logged on today (late morning) the sites I needed to be looking at were okay; you know -- "important" ones: isellpens, Goulet Pens, Ebay.... :lticaptd:

Ruth Morrisson aka inkstainedruth


"It's very nice, but frankly, when I signed that list for a P-51, what I had in mind was a fountain pen."

#6 notbob

notbob

    Vintage

  • Member - Gold

  • PipPipPipPipPipPipPip
  • 407 posts
  • Flag:

Posted 30 May 2013 - 03:06

I'm no sysadmin and I don't pretend to tell you folks your business, but FPN definitely needs to start looking for another hosting service.  Seriously. 

 

BTW, this message editor is pretty messed up.  It took me 3 tries to change the font of this msg to Georgia font and then when I finally looked at the msg preview, the editing function crashed.   Things are far from back to normal.   


nulla dies sine linea

#7 lapis

lapis

    medium rare

  • Member - Gold

  • PipPipPipPipPipPipPipPipPipPipPip
  • 9,733 posts
  • Location:West Berlin
  • Flag:

Posted 30 May 2013 - 08:26

Thanks from here as well. Personally, I don't think it matters so much who is responsible for the outage, it matters so much that our admins have the knowledge necessary, and can and will spend their time and energy to fix things!

 

Fritz


Life is too short to drink bad wine (Goethe)

#8 OakIris

OakIris

    Antique

  • Member - Gold

  • PipPipPipPipPipPipPipPipPip
  • 1,656 posts
  • Location:Colorado
  • Flag:

Posted 30 May 2013 - 11:58

I wondered what happened; none of the other sites I frequent had a problem yesterday so I just figured that FPN was the "victim" once again.  This time though, the site seemed to have totally disappeared, so that was worrying!

 

Glad to see the site is back up, but I must say that, thus far, 2013 has not been good for FPN.  :(

 

Holly



#9 jacksterp

jacksterp

    Vintage

  • Member - Gold

  • PipPipPipPipPipPipPip
  • 400 posts
  • Location:NC
  • Flag:

Posted 30 May 2013 - 15:11

Happy for all things are back to "normal."


Fool me once, shame on you.
Fool me twice; damn
There goes that fox again.

#10 ethernautrix

ethernautrix

    Be the Pen Posse you want to see in the world.

  • Member - Gold

  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 10,655 posts
  • Location:Cloud Eight

Posted 30 May 2013 - 16:32

Heh. Didn't notice. Every time I came over here, here was here. :-)


I ink; therefore, I am
Fleekair <--French accent.

#11 OakIris

OakIris

    Antique

  • Member - Gold

  • PipPipPipPipPipPipPipPipPip
  • 1,656 posts
  • Location:Colorado
  • Flag:

Posted 30 May 2013 - 17:18

Heh. Didn't notice. Every time I came over here, here was here. :-)

lol - well then, you were lucky!  Every time I tried to get here yesterday, here wasn't here, here didn't exist!   :P  (Luckily, I was able to go there instead....)

 

(I always enjoy seeing what new avatar you come up with, Lisa - your current one makes me chuckle!)

 

Holly


Edited by OakIris, 30 May 2013 - 17:20.


#12 RLTodd

RLTodd

    Thor's Legions

  • Member - Gold

  • PipPipPipPipPipPipPipPipPipPip
  • 3,218 posts
  • Flag:

Posted 30 May 2013 - 17:39

Oh, gosh, this is just short of awful.  ( I have to keep perspective because there are always things happening in the world that are really awful, like wars & plagues & floods & such.)

 

I keep thinking of that old thing, "For want of a nail...."  Or, House of Cards, or Jumanji for younger folks.

 

I wonder if our systems, some of which we depend on to keep living, have become so complex that one pin pull out somewhere along the line collapses the whole thing?

 

I am not a believer in coincidence.

 

I am not in panic mode, but I am beginning to get a bit worried.


YMMV

#13 wimg

wimg

    Stip Etruria nut :)

  • Admin

  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 18,736 posts
  • Location:Maastricht, Netherlands, EU
  • Flag:

Posted 30 May 2013 - 20:44

Well, we got an official apology today.

 

However, it appears their cloud systems were ok, as they were on another part of the infrastructure, with their own internet connection, and had priority access to the internet backbone as well - they should not have mentioned that in their email, as that is really upsetting.

 

Considering what has been going on for the last three months, since they migrated to a new data center, which, BTW, is still being built up, we already had been looking into alternatives.The problem is that a move to another host provider will mean down time, likely for a few days, considering the amount of stuff to be migrated across. A full backup currently takes 5 whole days to download (!), and an upload probably 10X as long (ouch), so we would need to find a solution for that as well.

 

The team they have which is looking after us is absolutely fantastic, but if they don't have the tools to help us, it is going to stop somewhere. Also, all of these problems have cut a HUGE dent in our budget, as we had to get in a few (paid) consultants, and also had to temporarily get another server.

 

So, I guess we will be moving, once we find a solution to migrating all data across in a reasonably fast fashion. We've got some ideas as to the approach to take - need to work this out further, though.

 

Warm regards, Wim

 

 


the Mad Dutchman
laugh a little, love a little, live a lot; laugh a lot, love a lot, live forever


#14 wimg

wimg

    Stip Etruria nut :)

  • Admin

  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 18,736 posts
  • Location:Maastricht, Netherlands, EU
  • Flag:

Posted 30 May 2013 - 20:59

Oh, gosh, this is just short of awful.  ( I have to keep perspective because there are always things happening in the world that are really awful, like wars & plagues & floods & such.)

 

I keep thinking of that old thing, "For want of a nail...."  Or, House of Cards, or Jumanji for younger folks.

 

I wonder if our systems, some of which we depend on to keep living, have become so complex that one pin pull out somewhere along the line collapses the whole thing?

 

I am not a believer in coincidence.

 

I am not in panic mode, but I am beginning to get a bit worried.

It appears something somehow went wrong with the ip configuration tables in some very highly specialized routers, and not just one, but a whole bunch of them as used by the peer network provider, one which is one of the top 5 biggest peer network providers worldwide, so yes, I am getting worried too.

 

For those not in the know: this is about access to the (internet) backbone, which essentially is the internet super highway. Now, all internet traffic congregates from, so to speak, internet dirt roads, provincial roads, state roads, villages, towns, cities, mega-cities and metropolises right onto the internet highways, before being moved along to the internet super highways. What went wrong here is that a whole bunch of equipment controlling access from the internet highways onto the super highways decided to block all traffic, instantaneously and simultaneously, yesterday morning around 7 am EDT. And this could only be fixed about 12 hours later. 

 

Yeah, worrying indeed.

 

Warm regards, Wim


the Mad Dutchman
laugh a little, love a little, live a lot; laugh a lot, love a lot, live forever