BA and it's IT systems problems

[Mr Quatro]

I got to wondering how such a giant in the aircraft industry could have so many problems: http://www.reuters.com/article/us-britain-airports-heathrow-idUSKBN18P01O

Then I got to the bottom of the article ... other giants should be aware.

The GMB union said that BA's IT systems had shortcomings after they made a number of staff redundant and shifted their work to India in 2016.

"This could have all been avoided. BA in 2016 made hundreds of dedicated and loyal IT staff redundant and outsourced the work to India," Mick Rix, GMB National Officer for Aviation, said.

Plus BA started charging for snacks :yep:

[STEED]

Running Windows 10, too much info going back to Microsoft caused a crash. :roll:

[Jimbuna]

BA chief executive Alex Cruz says he will not resign and that flight disruption had nothing to do with cutting costs.

http://www.bbc.co.uk/news/uk-40083778

[Catfish]

BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said

So? Power surge. What has that to do with Win 10?

This could have all been avoided. BA in 2016 made hundreds of dedicated and loyal IT staff redundant and outsourced the work to India," Mick Rix, GMB National Officer for Aviation, said

I take it inviting workers to England and pay them better was out of the question.

[Rockstar]

Shut down could also be related to the recent religious fervor.

[McBeck]

I have worked in IT outsourcing and offshoring since 2009 and moving IT to India is in itself not a problem anymore than it is moving it to any other country. Our IT service quality and customer satisfaction rose with the move...

We can only speculate into if that move made any impact.

No, what I wonder about is the power surge problem. Any serious business with a 24/7 availability requirement must build critical IT systems with redundans.
That means spreading production systems over more than one datacenter and with different powersources and hooking up to separate powergrids.

If power was indeed the problem, then BA has a MUCH bigger and fundamental IT design problem...offshoring is insignificant in that perspective.

[Jimbuna]

I have worked in IT outsourcing and offshoring since 2009 and moving IT to India is in itself not a problem anymore than it is moving it to any other country. Our IT service quality and customer satisfaction rose with the move...

We can only speculate into if that move made any impact.

No, what I wonder about is the power surge problem. Any serious business with a 24/7 availability requirement must build critical IT systems with redundans.
That means spreading production systems over more than one datacenter and with different powersources and hooking up to separate powergrids.

If power was indeed the problem, then BA has a MUCH bigger and fundamental IT design problem...offshoring is insignificant in that perspective.

Well put matey :yep:

[ikalugin]

One must account for the slim possiblity that redundant power sources went out at the same time because even with extensive redundancies you dont get 100 percent availiability.

[McBeck]

One must account for the slim possiblity that redundant power sources went out at the same time because even with extensive redundancies you dont get 100 percent availiability.
Very doubtfull, as the consequence would have been that both datacenters connected powergrids would have failed. Datacenters have redudancy by connecting to 2 powergrids (at least), so min. 4 powergrids that have homes, other businesses etc would have been down. If that was the case I think it would have made major news :)

Datacenters and high availability systems have redundans built-in on many levels. Its not very uncommon that redundans is activated on storage, network or other levels. It takes a disastor to impact power on datacenters to the point where both fails.

Back in the day when I was involved in Disastor tests we joked with the scenario of both datacenters failing, because if that was the case pur country would be in a disastor situation :)

[Mr Quatro]

You would be a good man to have on anyones think tank boardroom, Mc Beck :up:

If you know the location of anyones power source and back up power source ... then all you would need is a sleeper cell or two with those new hand held EMP guns to disrupt your enemy, right?

The future of warfare just gets more compilcated.:hmmm:

[McBeck]

You would be a good man to have on anyones think tank boardroom, Mc Beck :up:

If you know the location of anyones power source and back up power source ... then all you would need is a sleeper cell or two with those new hand held EMP guns to disrupt your enemy, right?

The future of warfare just gets more compilcated.:hmmm:
Well, we also joked about the actual fact, that if you really wanted to perform an attack that impact a country, target whole datacenters that act as eachothers failover. Chances are that this could actually cripple/kill companies and even impact whole countries. Just think taking out datacenters that hosts top 2 banks in a country + a few central big business.

[Aktungbby]

I beat the bullet by a month apparently; I flew BA to London Gatwick and missed the connection to Edinburgh due to an airline delay in Oakland. We had to be bussed at airline expanse to over to Heathrow for the next connection. A computer problem would have been miserable.:k_confused:...and expensive as I love the duty free shops at the airport!:O:

[ikalugin]

Very doubtfull, as the consequence would have been that both datacenters connected powergrids would have failed. Datacenters have redudancy by connecting to 2 powergrids (at least), so min. 4 powergrids that have homes, other businesses etc would have been down. If that was the case I think it would have made major news :)

Datacenters and high availability systems have redundans built-in on many levels. Its not very uncommon that redundans is activated on storage, network or other levels. It takes a disastor to impact power on datacenters to the point where both fails.

Back in the day when I was involved in Disastor tests we joked with the scenario of both datacenters failing, because if that was the case pur country would be in a disastor situation :)
Look at it from another direction - 99.9 percent availiable datacenter would still be unavailiable 1 day per 3 years. Despite how improbable it may seem that a datacenter would fail, especially to a specific cause (power), between the many datacenters it would occur fairly regularly.

p.s. my view is formed by operations of a major telecom and other services provider.

[Catfish]

This is all right and perfect, but as McBeck and Ikalugin say, a 100 percent security is almost impossible to reach, and pay for.
You get 99 percent ok, but to have 99,9 percent the costs already rise exponentially from 99. Failover clusters are all nice unless you have a real big and lasting power cut.
I take it that BA's server farms are distributed all over the UK, and also abroad. But if the ethernet connection fails (routers need electrical current, too), it does not work at places where the power fail happens.

Also the data synchonizing is a real headache already when all works, with Windows. Even if branch cache and failover clusters work perfectly, power backups like Diesels only work locally and temporarily, and will not support all the routers between e.g. London and Liverpool.

The urge to control all digitally makes us very vulnerable, apart from also giving up personal privacy. Marc Elsberg describes that very well in bis book "Blackout".

[STEED]

Willie Walsh, the head of British Airways owner IAG, has backed the BA boss in his first interview since the IT fiasco that grounded flights last weekend.http://www.bbc.co.uk/news/business-40117382

[ikalugin]

Fairly recently a major cell phone operator (Megafon) simultaneously lost both its primary and backup user data processing centers for Central/Southern region. Needless to say they had some service issues until they managed to migrate their data to a makeshift solution.

[Jimbuna]

http://www.bbc.co.uk/news/business-40117382

Both members of the same Old Boys Club :03:

[Mr Quatro]

Well, we also joked about the actual fact, that if you really wanted to perform an attack that impact a country, target whole datacenters that act as each others failover. Chances are that this could actually cripple/kill companies and even impact whole countries. Just think taking out datacenters that hosts top 2 banks in a country + a few central big business.

I don't want to make a big deal out of this, due to my lack of understanding on what a data center is, but it seems very important to most western countries.

How long if ever would it take to find out if it was an accidental power surge or an enemy induced power surge? You can't very well go to war with a suspicious third world country when in reality it was a smarter well informed agent of evil wanting to get even for who knows what?

[McBeck]

And they are calling in the scapegoats!!

http://www.dailymail.co.uk/news/article-4565236/IT-engineer-blame-BA-s-150million-global-meltdown.html

If the business takes such an impact from a 15min outage then they have designed the system incorrectly. That outage could have come from many other sources where they couldnt call it human error.

[McBeck]

I don't want to make a big deal out of this, due to my lack of understanding on what a data center is, but it seems very important to most western countries.

How long if ever would it take to find out if it was an accidental power surge or an enemy induced power surge? You can't very well go to war with a suspicious third world country when in reality it was a smarter well informed agent of evil wanting to get even for who knows what?

They are important, but do you think a CIO would pay attention to such a risk? Security has always been difficult to explain the value of because you only experience it for real when it fails and when it fails the impact can be rather high. High impact and very small chance risks are not easy for people to deal with.

Fact of the matter is that datacenters are not fortresses and can be physically attacked.