What Happened?

[oddoneout]

Never understood how UPS’s (uninterruptible power supplies ) end up being interrupted. In a well known broadcasting organisation generators would kick in when mains went . Not so much of a problem in the analogue days - the Audio desks would go into low power mode , various lights wouldn’t work .From what I remember vision desks would go into emergency cut mode. But digits don’t like it .Some Key places even had battery rooms Going back to WW 2. It was basic but it all went out.

Weren't similar questions asked when Heathrow had a massive outage earlier this year?
Last year at work we were having a lot of problems with our communications- even by our standards where interruptions and drop outs are pretty routine, due in no small measure to a rather flaky power supply to the site as a whole - and various checks, remote and otherwise, couldn't come up with a reason. I happened to go into the comms room(actually glorified cupboard) on the hunt for a missing vacuum cleaner and noticed that the stack of gubbins was all lit up with flashing red lights and distress messages on the UPS rack. I relayed the info and an emergency callout came and dealt with the dead and moribund items. Turned out that a member of the office staff who should have been doing the weekly checks in there hadn't bothered for rather a long time... Not sure why the 'support' at head office didn't suggest going into the comms room in the first place. Perhaps they weren't aware of the apparent limits to the checks they were doing remotely?
Anyway, so glad to have the forum back - I hadn't realised just how important a part of my day it has now become.

[DracoM]

<< I hadn't realised just how important a part of my day it has now become. >>

Absolutely. Well put!

[Ein Heldenleben]

Weren't similar questions asked when Heathrow had a massive outage earlier this year?
Last year at work we were having a lot of problems with our communications- even by our standards where interruptions and drop outs are pretty routine, due in no small measure to a rather flaky power supply to the site as a whole - and various checks, remote and otherwise, couldn't come up with a reason. I happened to go into the comms room(actually glorified cupboard) on the hunt for a missing vacuum cleaner and noticed that the stack of gubbins was all lit up with flashing red lights and distress messages on the UPS rack. I relayed the info and an emergency callout came and dealt with the dead and moribund items. Turned out that a member of the office staff who should have been doing the weekly checks in there hadn't bothered for rather a long time... Not sure why the 'support' at head office didn't suggest going into the comms room in the first place. Perhaps they weren't aware of the apparent limits to the checks they were doing remotely?
Anyway, so glad to have the forum back - I hadn't realised just how important a part of my day it has now become.

Yep if memory serves UPS systems are supposed to be checked weekly and tested monthly (at non critical times of day - sometimes the test trips things ) . One problem with battery rooms used to be oxidation.

[Dave2002]

Or were they Trumped?

Russian or Chinese interference - or perhaps Iranian - with the US election?

[gurnemanz]

<< I hadn't realised just how important a part of my day it has now become. >>

Absolutely. Well put!

Or as Joni put it: You don't know what you've got till it's gone?

[french frank]

Never understood how UPS’s (uninterruptible power supplies ) end up being interrupted. In a well known broadcasting organisation generators would kick in when mains went .

Yes, if I understand it, that is what happened. But the problem was the 'unclean shutdowns' of all their servers 'in particular our SAN file systems ... have corrupted "changelog" meta files that are critical for proper operation', whatever SAN file systems are. The first time it went down I was getting the 'server cannot be found' message, but the second time, after they'd been working on the problems, the message was 'database error'. So the recovery process is gradual, and the email system was left until last (and still isn't functioning).

[Serial_Apologist]

Or as Joni put it: You don't know what you've got till it's gone?

That's why I keep all my old VHS tapes! - and luckily something still working to play them on!

[jayne lee wilson]

Or as Joni put it: You don't know what you've got till it's gone?

I think the problem can sometimes be that you do!

Relieved to see the Friends-in-Music back again....

[teamsaint]

That's why I keep all my old VHS tapes! - and luckily something still working to play them on!

Thats two of us, S-A

[Ein Heldenleben]

Yes, if I understand it, that is what happened. But the problem was the 'unclean shutdowns' of all their servers 'in particular our SAN file systems ... have corrupted "changelog" meta files that are critical for proper operation', whatever SAN file systems are. The first time it went down I was getting the 'server cannot be found' message, but the second time, after they'd been working on the problems, the message was 'database error'. So the recovery process is gradual, and the email system was left until last (and still isn't functioning).

I think it’s a shared area network which cleverly links storage across multiple servers . I guess it’s supposed to be more resilient than LANs or NAS. The fact that they have managed to piece it all back together maybe demonstrates that. Semiconductors don’t like power cut offs or surges ...

[french frank]

I think it’s a shared area network which cleverly links storage across multiple servers .

Don't understand much about servers, but that could explain why we've had to upgrade our package and shift from one server to another when traffic was a bit more ferocious than it is now.

[cmr_for3]

Great to see everything back. I had not noticed before how often I pop in/on for a browse! :)

[johnb]

Good to see everyone again. (I was wondering what was happening.)

[BBMmk2]

Great everything is back to normal! Could it have been a Trump/Halloween stunt? I hope not. Russians, hmmm ....
Glad that everything is back to normal. Was getting rather worried, to say the least. Well done our hosts!

[BBMmk2]

Relieved to see the Friends-in-Music back again....

So am I!