Friday 13 October 2017

Office DDE – How a zero day exploit evolves

12:28 Posted by G No comments
I’ve been following Kevin Beaumont on Twitter for a while, he’s a security architect from Liverpool, and quite a regular blogger.

On Tuesday lunchtime I read a tweet from him :

Having read the article at Sensepost, they have discovered a way to run code (what is called in IT security nerd circles as ‘RCE – Remote Code Execution’) in Office documents without the use of Macros.  Macros are a traditional way of getting malware to run, but companies often block macros, and some anti-virus services are configured to remove them, so not a reliable way to get malware on your victim’s machine.  Using this DDE feature is a newer and easier way to potentially deliver malware.

Clearly this is a big deal.  Reading the full post from Sensepost, they have reported it to Microsoft and Microsoft have said this is expected behaviour and therefore they won’t be patching it.  This means that customers are vulnerable to this attack vector and will have to find another way to protect themselves.

The main point of this article is to illustrate how quickly things move and this threat evolves.  From the Sensepost article no anti-virus spotted this as a suspicious file.

Beaumont then goes on to test this himself, being able to create a word document that will start the calculator program from it, and showing that none of the malware protection running on his machine detect this exploit. This is now 6pm on Tuesday the 10th of October

By 7.30 that evening, one of the AV vendors has started to identify this type of file as malicious.

By 1am, Beaumont has discovered a word document that uses the DDE vulnerability to start Internet Explorer and open a website where the malicious code is stored.  What’s more interesting is that the site where the malware is stored is a US Government website (now shut down)

Then by 8am on the 11th, here’s a copy of the email which has the DDE Vulnerability embedded in it.

Finally at 5pm on Wednesday, there’s a write-up from Talos (part of Cisco) about the whole malware chain

What’s also interesting is that the hackers use DNS to exfiltrate data, which is quite an esoteric way of doing it, but most companies won’t spot it as DNS is a perfectly legitimate service to have running.


If you managed to follow this to the end, this is clearly a very sophisticated hacking attempt, here’s the core elements of campaign :
1. Use of DDE exploit, which is not commonly known, and won’t be patched by Microsoft
2. Lack of Anti-virus firms picking up this attack vector
3. Use of legit sounding emails (Purporting to be from the SEC relating to EDGAR (company filing system in the US)
4. Malware is downloaded from a legit US Government website

This all goes to show how sophisticated attackers are, and how important it is to stay vigilant

In subsequent blog post Beaumont goes on the make the point that Microsoft will have to do something about this, as it is so difficult to protect against.  His suggestion which makes good sense to me is to disable DDE by default, and enable it via a registry key.



Wednesday 21 June 2017

Prodrive, Le Mans 2017 and Social Media done right

10:56 Posted by G No comments
For the first time in a few years, I managed to watch a reasonable amount of the the Le Mans coverage this year.  I love a bit of endurance racing, and clearly to us Europeans Le Mans is the biggest race of the year.  I've still never been, something I need to rectify in the very near future !

I missed the actual finish, but via a few friends on social media I was aware of how exciting the finish was in the GTE pro class (spoiler alert) with the Covette and the Aston battling it out right to the very end.  I find it astonishing that after 23 hours 55 minutes and 2,800 miles of hard racing the cars were less than a second apart.

I saw this morning on Prodrive's Facebook page they'd posted the last 5 minutes of the race, plus some post race celebrations.  It's makes for great watching (especially if you're an Aston fan!).



What I really liked though is the interaction between the Prodrive team and the punters on Facebook. Clearly it must have been gut wrenching for the unfortunate Covette team, but I like the comment from the Prodrive team saying that they went and spoke to the Covette team afterwards.  I like that it's a two way conversation with the Facebook followers, and also that even in this highly competitive, big budget world the team still have the humility to go and talk to their competitors

Thats the spirit that all racing should aim to emulate, F1 has much to learn both from a racing spectacle as well as fan interaction.

Here's the video :


Monday 5 June 2017

Why BA should care about IT

20:42 Posted by G 1 comment


Having been in the eye of the storm of the BA IT systems failure last weekend, and only getting away on holiday, 2 days after we should have, I think there’s lots of things to learn.


I think what most struck me about the outage was the sheer size of it.  Upon arriving at Heathrow terminal 5 on Saturday morning with the extended family all excited about a week’s holiday in Greece, we were met with huge queues outside T5, and at that stage it looked like a baggage or check in problem. But then over the course of the next hour it quickly became clear how severe the outage was.  Not only were check-in systems not working, but the departures information boards had been stuck since 9.30 am.  Even when we got to the gate, which turned out to be the wrong one, there were planes on stand waiting to push back, more aircraft waiting for a gate, and flight crew equally confused.  When we did finally get on board an aircraft the pilot informed us that the flight planning systems weren’t working so he couldn’t create a flight plan, and therefore was unable to work out the correct amount of fuel to put on board, and without that he was unwilling to push back off the stand.  Even when we got the news (first via the BBC) that all flights were cancelled, the pilot told us even the system to cancel flights wasn’t working.  This meant that it getting busses to take us back to the terminal took a long time, followed by the ignominy of having to go back through passport control having not left the airport let alone the country.

From an IT perspective there’s a few interesting aspects.  Firstly BA have claimed this to be a power related incident.  This is an interesting cause.  As far as I’m aware there were no other companies impacted by this outage, which strongly suggests that this is was not in a shared (co-located) data centre, as otherwise we’d have seen other outages.  This also implies that BA aren’t running in the cloud, as we saw no cloud outages over the weekend.  Secondly assuming this was a dedicated BA data centre then there’s been a major failure of resiliency.  I would normally expect of any decent quality data centre that there would be a battery backup to provide power in the immediate follow-up to a power failure.  As soon as there’s been a power failure detected diesel generators should kick in to provide longer term power.  Normally batteries would sit in-line with the external power to smooth the supply and provide instant protection if the external power fails.  At this level of criticality it would be normal to have 2 diverse and ideally separate power suppliers.  The diesel generators are some of the most loved engines in the world, they are often encased in permanently warmed enclosures to keep them at the correct operating temperature.  Quite often the diesel they consume is pre-warmed as well.  This often also is stored in 2 different locations to ensure that if one gets contaminated there’s a secondary supply that can still be used. These engines are often over a million pounds each, and in some sites I’ve seen then have n+n redundancy (if 4 generators are needed there are 8 on site) to deal with 100% failure.  Clearly as a customer you pay more to have this level of redundancy but as we’ve seen over the weekend you never want an incident like this.

In addition to having all this redundancy built into a data centre its vital that all these components are regularly tested.  It’s normal for data centres to test battery back-up and run up the generators at least once a month to ensure all the hardware and processes work as they should in an emergency.

Once you’re inside the data centre, all the racks (where servers are housed) are typically dual powered from different backup batteries, and power supplies, and then each server is dual powered to further protect against individual failures.  In total there are 6 layers of redundancy in between power coming into the data centre and the actual server (redundant Power suppliers, redundant Battery back-up,  redundant power generators, dual power to the rack, dual power supplies to the server, redundant power supplies in the server itself).

As you can see in theory it’s pretty difficult to have a serious power failure.  While it’s possible to have a serious failure in parts of a power supply system, it would be highly unusual for this to be service impacting.

However as we saw in the outage at the weekend something catastrophic must have happened to produce such a widespread outage, and one that seems to have affected BA globally.

Even outside of pure power redundancy most large corporations will have redundancy built into individual systems, be that within the same data centre or in a secondary site (ideally both).  For the more sophisticated sites, these are often what’s known as active-active, i.e. the service is running in both sites at the same time, so if there’s a failure in one server or site the service keeps running but with degraded capacity (the application may appear slower to users), however it is still available.

Most companies will spend at least 7 figures sums annually running with this level of redundancy and will test it regularly (most regulators insist this is at least every two years).  It would appear that for this level of outage and number of systems that failed, either there wasn’t the appropriate level of redundancy or it hasn’t been tested regularly enough.

It’s worth pointing out that all the points mentioned above are expensive, painful to test, and do little to add to the bottom line of the company, but it is just this sort of ‘insurance’ that you never want to rely on, but having thorough and well tested plans makes all the difference when this sort of event happens.

 There’s been lots of reports in the UK press, and comments from unions saying this event is reflective of BA outsourcing its IT services to a third party.  I’m not sure if outsourcing had any impact on the outage, but the mere fact that if BA do outsource their IT it’s an indication that they do not perceive IT to be a core function for BA, as they’ve asked someone else to do it on their behalf.

You may have read many IT articles about Uber being the biggest taxi company and owning no taxis, and airBnB being the biggest hotel chain, but owns no hotels.  It’s clear that both these examples are technology companies not traditional taxi or hotel vendors and therefore with such a reliance on technology they would be expected to have very highly resilient systems that are regularly tested. 

BA however doesn’t fit that model, their biggest expenses wouldn’t be IT, they probably spend significantly more on aircraft, fuel, staff etc.  However when I thought about it, their main systemic risk probably is IT.  If any one model of aircraft was grounded for any reason they use a range of planes in their fleet so this would be impactful, but not catastrophic.  Similarly if one of the unions that some of their staff belong to goes on strike (as we’ve seen in the past) is annoying but not critical.  The same could probably be said for their food or fuel vendors, who probably vary around the world, and so if any one of them fail, they can most likely work around an individual failure.

Not so with IT, it appears that one power failure in one data had the ability to completely cripple one of the biggest airlines in the world.  I cannot believe that BA would have actively known this risk and chosen to run with it.

It the ever increasing digital world we live in every company is slowly turning into a technology company.  Maybe not front facing, but even in a traditional industry such as aviation where aircraft hardware will always be key, this weekend proved you can have all the planes in the world, but if the tech isn’t there to support it, you’ve got no business.



Thursday 9 February 2017

The cloud gets serious

18:32 Posted by G 1 comment
I've been interested to read today, about the announcements from Snap and also Evernote.  For those that have read other blog posts of mine, you'll know I'm a big fan of public cloud, and can't see the point of private cloud.  For us as a medium sized business (2,500 staff) running our own data centres in the medium and long term makes no sense, this is pure undifferentiated heavy lifting, and something that Amazon and Microsoft can do at a scale and cost that we can't (and shouldn't) compete with.

However my understanding was that for the really big boys (Facebook, Twitter etc.) running in public cloud was more expensive that running their own data centres.  Well the announcements today, Snap having an eye watering $2bn deal (over 2 years!) with Google cloud, and an addition $1bn over 5 with Amazon clearly says 'we don't want to be in the data centre game'.  Here's the Silicon Angle article with more detail - link

Evernote have just published an article saying they've moved 3 petabytes to Google cloud, and are now out of their own data centres, impressively in 70 days, here's their blog post - link

Fascinating to see Google with these two big wins both announced within a day of each other, is this the tipping point when no one runs their own data centre any more ?




Thursday 12 January 2017

Best story ever - Lord Robert Winston

18:48 Posted by G No comments
I've just been to our Company's UK annual kick off, our theme this year being adaptability, and the Leon Megginson quote (often wrongly attributed to Charles Darwin) of :

It is not the most intellectual of the species that survives; it is not the strongest that survives; but the species that survives is the one that is able to adapt to and to adjust best to the changing environment in which it finds itself

One of the more fun elements of the launch event is the guest speaker, we've had Sir Clive Woodward talk about marginal gains (and a fascinating video story or preparing for every eventuality), Ben Hunt-Davis (GB Olympic gold medal winning rower), who talked about making the boat go faster, and how they prepared for their Olympic gold medal race.

However today's speaker was Lord Robert Winston (or to give him his full title - Robert Maurice Lipson Winston, Baron Winston FMedSci FRSA FRCP FRCOG FREng), who was a fascinating presenter, I thought his presentation covered a huge range of topics, but all fascinating.  The way he intertwined history, genetics, maths, physics and classical music, shows the size of his brain, which was one of topics he talked about.

He told the story of Albert Einstein's autopsy, when his brain was examined his brain was found to be no different to anyone else, and he made two points, one to emphasise this to our children, to tell them there's nothing physiological holding them back from being the next Einstein, but also that (while pointing to two random people in the front row), that between then they have more brian power than Einstein, and that was the power of collaboration.

But for me the most powerful thing he said was in answer to a question asked from the audience.  When asked what he was most proud of, he said (without missing a beat) was he was most proud of his kids, which as someone who has done more for fertility in the last 40 years that probably anyone else in the UK is an understandable answer.  But when pushed (and in a typically very modest English way that only people who have really achieved big things can), said...

" I met  a young lady at an event, who had waited patiently to speak to me after the event, who told me her name, which didn't immediately register with me, but when she explained her story she broke into tears, as she explained that I had modified her genes (apologies I've got the details wrong here) to remove a genetic mutation that killed her brother, and as we spoke I broke into tears, telling here when I did the pioneering operation on you as an embryo that was the first in the world and didn't become a standard procedure for another year or so, and the politicians of the day wanted to ban the operation, which would have meant you would not have lived"


I was stunned with this story of his humility, his pride and his general all round goodness, and it was just a real privilege to listen to him speak, can we clone him please ?

Without a doubt a great story, and possibly the best 'what are you most proud of' anecdote - Lord Winston I salute you !

Friday 6 January 2017

Delegate to the edge

19:59 Posted by G No comments
One of the best books I read in 2016 was Matthew Syed's - Black Box Thinking. As an IT leader reading him talk about feedback loops, the use of Agile, failing fast and learning from our mistakes was really a message that I am used to delivering. It's a really engaging book, with a really gripping (but tragic) first chapter. I was lucky enough to speak to him at a Deloitte event and got to tell him this, but it really draws you into the book. I can't recommend it enough.


Since finishing the book, I'm an avid follower of his on Twitter, and from his Twitter feed, I noticed that he recently wrote an article on LinkedIn about delegation and the power of it.  What struck me most (apart from delegation being the only way to actually get stuff done!) was using the priviledge of using our position as leaders to empower our team to get stuff done.

What I mean by that is my job is to clear the path, set long term direction and gain budget approval, but the real brains of the outfit are those that work for me, they come up with how we are actually going to achieve our goals, and executing them.

As Syed explains in his article (here), that empowering teams, and pushing decision making closer to the 'edge' of the organisation is much more effective.  I'm currently reading General Stanley McChrystal's book which delves into this and is fascinating, as clearly in his domain there's much more at stake than mine.

What is closer to home (especially as an avid hockey play) is the distinction that he draws between the England Football team and the GB (Olympic gold medal winning) Women's hockey team. Whereas the England football team looked listless while losing to Iceland (a central command and control structure - without the manager there was little leadership), to that of Danny Kerry the GB Women's coach who is quoted as :

Kerry allowed players to decide on when they train each day, codes of conduct, and they elect their captain through a vote. What happened? The players developed leadership qualities, and felt far more empowered to make big decisions on the pitch.

The article goes on to quote Eddie Jones the England Rugby coach, who says:

Eddie Jones, has become interested in the idea of “growth mindset”, trying to ensure that his players are willing to take responsibility for their actions, rather than making excuses when things go wrong. 

Which neatly joined the dots with an article that Sir Clive Woodward (England Rugby coach 1997-2004).  In an article posted today in the Mail, Jones is quoted as saying:

The only advantage you really have on the opposition is learning faster, so if you want a learning environment the head coach has got to set the example. To achieve that you have to keep improving yourself, keep gaining knowledge wherever you can and you’ve got to have a coaching staff the same

Woodward then asks : With the ideas, how much is coming from you and the coaches as opposed to thoughts and ideas from your team?

Jones's answer is : The balance probably went from 100-0 in favour of the coaches 12 months ago, now it’s probably 50-50 and we want it to be 20-80 by the World Cup.

Lots for us all to learn, but a mix of delegated leadership and delivery accountability at the edge makes good sense to me

Delivering Technology in the new digital world

09:22 Posted by G No comments
Digital, you can't move for people talking about digital, but this article in the Wall St Journal grabbed my attention. 


I like the Equifax and Liberty Mutual approach, that the command and control structures that we are used to are breaking down and new cross functional teams are being formed, and we need to adapt how we deliver technology to them.

 This quote from their CIO Dave Webb is spot on in my opinion

“Companies that transition from a more traditional business model to one built around technology and IT must adopt a management practice that can support this evolution,” says Dave Webb, global chief information officer of Equifax Inc. 

The article goes on to say

Equifax in 2015 opened a technology-development facility in Auburn, Ala., the facility, which develops automation and platform services is organized into small, cross-functional teams made up of a mix of business and technology personnel. The managers include a vice president of global platform engineering, an automation lead and a cloud and systems lead.

Rather than issue top-down directives, these managers instead strive to help self- directed teams leverage digitally enabled data sources, collaboration and sharing tools, and tighter feedback loops, to “get things out the door faster,” Mr. Webb says. 

Finally the Liberty Mutual CIO James McGlennon adds :

At Liberty Mutual Insurance Co., collaboration and sharing tools powered by digital capabilities—roughly 80% of its operations currently run in the private and public cloud—are bringing business, sales and IT units together like never before, says CIO James McGlennon.

That’s shifting the role of managers who oversee those units away from “dictating how things should be done,” to acting more like coaches who guide collaborative, multifunctional teams to “get the work done” on their own, Mr. McGlennon says.

Both these CIO's are doing what we're trying to do, in trying to create coordination teams at the centre to orchestrate and cultivate communities, but with delivery at the edge where the situational awareness is best

Thursday 5 January 2017

Situational awareness at the edge

19:56 Posted by G No comments
I first saw this tweet from David Bray, CIO of the FCC : 

and I thought that made good sense. I found the article that it came from which is here : https://www.rainpartners.com/secrets-of-it-leadership-a-conversation-with-dr-david-bray-senior-executive-and-cio-of-the-fcc/ and it makes for interesting reading. I liked his comments :

"As a CIO I think the best thing I do is listen, and learn, and try to build a case as far as what are the different things needed for the different offices.

The second thing is actually to be very open that you want people to give you feedback. In a changing environment the top is only a few people, whereas an organiation has many more people at the edge. And they’re actually going to know what best fits their context. So it’s much more about cultivating and raising their insights up, as opposed to just doing things from the top.

And the good thing is I’ve now been doing what my PhD showed now at the FCC, as well as with the defense and intelligence agencies, which is about championing change agents. Change agents, I give them autonomy, I give them a measurable sense of progress, and I give them a meaningful source of work, and that intrinsically motivates them to be much more doers and getting things done at the edge, than it would ever be possible if we tried to do it in terms of command and control from the top."

and

"Another thing is you not only have to empower the edge, but you also have to encourage and cultivate a diversity of insights. I actually tell my team “I’m going to have blindspots”. I want them to be able to point them out. Bring data—so it’s not just your opinion—but I’m open at any given time. Come into my office, let’s have a conversation. And if you think there’s a better way we can go as an organization in terms of our strategy, please bring it.

I know I’m going to have blind spots, I’m only human, and we’re in an era of rapid, turbulent technology change, and so we all have to work together cultivating change agents."

Makes good sense to me