CTIO 101 Podcast

Ransomware Part 2: Backup vs Restore

September 22, 2022 Jon Grainger
CTIO 101 Podcast
Ransomware Part 2: Backup vs Restore
Show Notes Transcript

There is no doubt that backups are an essential part of any data management strategy, but equally important is the ability to restore data quickly and efficiently. Many IT professionals will argue that restore time is even more critical than backup time. Here are a few reasons why:

  1. Downtime is expensive: When data is lost or inaccessible, it can lead to costly downtime for businesses. The longer it takes to restore the data, the more money and productivity are lost.
  2. Customers demand fast recovery: In today's fast-paced business environment, customers expect near-instantaneous recovery times. If a system outage or data loss occurs, customers will require a quick resolution to avoid disruptions to their operations.
  3. Compliance requirements: Many industries have strict regulations around data retention and disaster recovery. If data cannot be restored quickly and accurately, it can put a business at risk of non-compliance and potential legal consequences.
  4. Reputation: In the event of a data loss or system outage, a business's reputation can be severely damaged. If customers lose trust in a company's ability to protect and recover their data, it can lead to long-term consequences for the business.

These factors highlight the importance of restoring time in a comprehensive data management strategy. While backups are essential, they are only one piece of the puzzle. To ensure that data can be fixed quickly and efficiently, businesses must invest in technologies and processes prioritising restore time as a critical component of their disaster recovery plans.

Rob:

at some point pretty early on, someone's gonna raise the question. Are we backed up? Can we recover? Cause that's, cuz if you've not got your files, that's gonna be your, your safety net. Right.

Malcom:

CTIO 1 O 1. Business Technology. Simplified and Shared Sponsored by Fairmont Recruitment, Hiring Technology Professionals Across the UK Europe Don't forget to subscribe!

Rob:

Um, and a lot of, a lot of people do with backups. It's pretty rare for companies not to have backups. There are, they are out there. It's amazing how many companies actually are relying on snapshots rather than actual backups. Um, you know, people who are still just prepared for, I guess the, the physical threats of fires, earthquakes, floods, power outages, that kind of thing. Um, but yeah, if you need to go and see if you've got a backup, you can recover from, and unfortunately. And the, the example I often use is, is if you think of a, going back to your film, analogies, Jon, if you take a, a classic sort of eighties horror film, there's a be in the house. And the, the slasher comes to knock on the door. The first thing he does is he, he cuts the phone line, right? Cause you take away, people's lifeline. You take away people's safety net and their, their ability to get help and get out the situation. And the same thing. They come into the network, they're sitting there for six months. They're, they're identifying where the backups are and they're taking them first because they know that's where you're gonna turn. Um, so you're gonna need to know when you, when you turn to your backups a, are they there?

Jon:

It's incredible

Rob:

yeah. It's scary. Cause you, you, you go to see them a lot of times you see them and you go, oh, the backups are still there. Great CEO, don't worry about it. I'll have us backup in, you know, eight hours, 10 hours a day, whatever your, whatever your recovery time is. Um, but how'd, you know that the backups are clean because what if you spend all that time recovering and the malware just spreads again, they lock you all down. And you go

Jon:

This technology, which claims to have an immutable backup, so a backup that cannot be changed. What you're saying is yeah. That backup cannot be changed, but what you've actually backed up has got the malware in it. Is that what you're saying?

Rob:

There's two route. So yes, you can, you, you might have backups that can't be changed, but if you've already, if they've already infected, then it's finding, it's finding a backup that isn't infected,

Jon:

So you want one that can't be changed that is also clean?

Rob:

Yeah. So there's, there's two of this. So there's, there's the element of finding a backup that's clean. There's also the element of, are there any backups left to recover from? Cause that would be the prime goal. The ransomware is if they can delete your backups or take them all, then you've got nothing to recover from.

Jon:

Yeah. So they, so, so you've got, you've got a backup, that's a mutable in the sense it can't be changed, but if they've got sufficient control over, it can be deleted. If you say immutability is a, is a, let's just say it's a reader, a write one CD rom. Yeah. I dunno why I've used that analogy, but we've put our day. Yeah. That's, we've burned it. That there's no way you can change that. So, whatever happens, we've got it. What they're doing is the equivalent of finding that and then breaking it or taking it away or burning it or, you know, they're actually, is that what they're doing? They're removing those,

Rob:

so the, yeah, there's, there's lots of ways you can attack an immutable backup. And this is why it's so infuriating to see immutability get chucked around as much as it, it does at the moment. Cause immutability is nothing new. It sounds really fancy and exciting and new, but like tapes, tapes are immutable. Um, it's just the, it's just the idea that this backup cannot be changed. But you know, as you say, if you've got a, if you have an immutable backup, but it sits on say a window server.

Jon:

Yes.

Rob:

if I can, if I can encrypt the server. Yeah. I can't change your backups, but guess what? You can't get to'em either

Jon:

So the CD ROM are, yeah, I'm speaking to you. I'm Rob. You're the CEO. Uh, I have no doubt, Rob. This will happen. Although I doubt you'll employ me, but let's just pretend you make CEO one day and you do employ me. I'll go Rob it. I'm really sorry. We have got the CD ROMs, first thing you'll be saying is Jon, why the hell are we using CD ROMs? And I'll say, well, just really good proven 1990s technology. Um, but they're in a safe, uh, and the ransomware folks have actually also got the key to the safe. We, we don't know how they did it. So the immutable copies are in there, but we can't get to them because they've the safe they've got the keys to. Is that, am I, am I gonna get in trouble with that analogy to,

Rob:

that, that spot on Jon. And guess what? You, you paid to get those keys back, don't you the first, the first thing,

Jon:

so I wanted to throw something. I wanted to throw something at, at this point, Rob, this is one of the things you're talking about immutability. So I've been in a scenario where I've had to restore, not through, uh, a ransom, but through a really, really significant nasty outage. And, um, I learned something really important, which is where you think, where you tell your business to store their data. Yeah. And where you set up to say, that's where the backup is. You know, we all use this system and that's where the backup and that's, that's where stuff goes. Isn't actually where everyone keeps their information. It's, you know, there's this saying, which is there's, there's many, many processes in a business. There's the, the process that the CEO thinks happens. There's the process that the it director reckons and there's the process that the quality person

Rob:

yeah.

Jon:

There's the actual process that happens on a Tuesday. There's the, the compliant, process's the process that was written down and they're all different,

Rob:

Mm-hmm

Jon:

know, that kind of approach. Well, I think the same with data. So a lot of companies will have a core system of some sort, some sort of enterprise system, uh, you know, whatever it is, it's the stated, this is how we're gonna operate. And as long as you make sure your activities are in this system, they will then get backed up. You know, regardless of the problems we've just described. If you don't have that, if you have systems that aren't compliant in the first place, then that data isn't gonna be in the backup.

Rob:

No. And

Jon:

place now that might coincidentally help you, but it probably won't because the whole reason why we're reaching for the backup is because everything else has just got blasted. Yeah.

Rob:

yeah. And remember, you know, you might have the unstructured data, you might have the backups to different places, but always remember these gangs have done their work. They've seen where everything goes and they, they know it. And

Jon:

And they've been there for ages.

Rob:

it. And you know, I don't know when, when you have to see this recovery, John, but obviously we're in a world now where people are working on sometimes personal devices, sometimes work devices. They're at home in the office. They have access to SAS apps. They have access to cloud storage.

Jon:

yeah,

Rob:

Data can get to a lot of places and a lot of people could be, can be unaware of what that data is, where it's going, how much done as you say. So just cause you've got that, that backup doesn't mean doesn't mean your, your, you know, your life raft is gonna even float. Going back to the threat of those backups, you know, we touched on the fact that you could just lock down the immutable backup, but there's other clever tricks they can do. So maybe you haven't got it set on a windows host that can be encrypted or whatnot, but maybe you haven't got MF. Maybe you got MFA on some of your critical applications, but you haven't got it on your backups. So they just get into your backups and they start playing around with maybe retention policies. You know, they just, they can't affect your backups, but they can just change those retention policies, make the backup device think it's been 30 days, 12 months, whatever it needs to do to wipe.

Jon:

which is, which is probably the, the Mo the cleverest way of doing it, because then you just get the policy to do the dirty work. Uh, and it might not even come up as being unusual because

Rob:

It, it wouldn't flag because it's, it's doing what it's

Jon:

very quietly. And then you just wait, and then you wait for enough gap between, you know, the last successful backup. And when you want to strike to then have that, that Corpus of information, that then is then vulnerable to the attack. Another one, Rob, to throw at. Is, um, you know, uh, the, the, we get a lot of great technology to, to, to, to work with, uh, and that we do make pretty serious claims about its efficacy. Uh, and no, no, I don't think anyone lies, but sometimes people get a bit lost in, uh, achieving the effectiveness of some software requires quite a lot of in depth knowledge configuration set up, you know, there's a context around it, et cetera. So is it also a case that with, with these sorts of technologies that you may have bought everything that you were meant to buy, but you didn't set it up. Right. You know, it's like a configuration issue. So you get, you know, someone in the infrastructure team is just, it's just gone very pale and said, well, it was meant to happen, but we, we haven't yet finished that part of the project. And actually we hadn't even switched on. I'm gonna say the immutable flag, but we we've already been there with that, but it could be some sort of form of protection or some, some element of it. Uh, which means that, um, we didn't even really have the protection in the first place. We had the potential of it, but we never got round to configuring it properly or we didn't understand it. is that

Rob:

Exactly. I, it comes down to the, the organization's approach and, and view of those backups. Are they, are they viewing those backups as just a, a business usual task they've got to do is just an operational task again. tick box exercise. Yes. We've backed up. We've done. Or are they treating this as a data protection exercise for if they need to, you know, invoke business continuity and

Jon:

see, yeah, we shouldn't be saying we've backed up. We should be saying, um, we're ready to restore. That should be literally the, the mind the mindset should be. Yeah. We're in a position to restore, not a yeah. Tick, uh, such and such, uh, has been backed up.

Rob:

No one, no one backs up to back up. You're only back up to recover. So why do you focus just on the backup piece? You should be focused on that

Jon:

Yeah, it's amazing. I mean, it happens everywhere, doesn't it? But that is a proper, uh, uh, example of meeting the target, but missing the point.

Rob:

Absolutely. And this is why it's so important to have these conversations early on in that, because if you are, if you work in, in it operations, as you say, whilst everyone's getting more prone and savvy to security requirements and, and these needs again, you've got different objectives. You've got, you've been given different KPIs, you've got different expectations of you. So you are viewing it right. You know, how can I make this as, as efficient as possible? Whereas if you can have a, a security team, look at it and go, right. Well, hold on. This is essentially our last line of defense there.

Jon:

Yeah. There's so much of, this is, um, this is about having a whole system approach. That's what this is about. Um, and, and I think, uh, kind of getting into the rehearsal space, having building a really, really positive relationship with between cyber operations, it operations, knowing what to do. Um, and I think, you know, having a mentality, which is about, yeah, we can restore rather than we've backed up. And, and I think also, you know, never, ever be, you know, be careful that you don't catch yourself about becoming complacent with a silver bullet.

Rob:

You have to ask those extra questions, additional things. It's not good enough to say, okay. ransomware hits. Have we got backups. Yes. Seems to go. Right. What if we couldn't access the backups? What if, you know, if we talk about the SMB market, if we're, you know, a smaller organization for a smaller team, what if, what if Jamie's on holiday? What if he's like, who, who does all the backups? And we can't reach that system. Like, have we got a single point of failure, all these different areas there? Like how do we,

Jon:

yeah, and, and again, you know, the folks, if they, if they're really in there, they might know, uh, when Jamie's, holiday's coming up

Rob:

it's probably set an outlook. They can see it and they go, right. Well, wait till, wait till here he

Jon:

Well, yeah. The access outlook via the web with no MFA in place, you know, you just it's. Yeah, it's incredible. Okay, cool. So where are we on the, um, on the scenarios at like three in the morning, we're all coffee'd out. We're all kind of, we're getting that point where you're not quite believe what's happened. Uh, you get the, you know, you get, you constantly get that moment. Like I said about it's, okay, we'll have a cup of tea and you turn the tap on. There's no water. There's a, there's a, that's happening a lot because you, you, a lot of your tech that you're using and increasingly the telephone, uh, with the other, with the, you know, with the, with the ransom folks on the other end, the phone is starting to become, uh, you know, it's catching your eye more and more because you are, you're sort of running out of, of, of runway.

Rob:

and they're gonna wanna apply the pressure as well. So I wouldn't be surprised if, you know, whilst you're trying to retrieve these backups or deal with your backup state or, or react to it, that deadline you got given maybe 72 hours, 24 hours, don't be surprised if that something gets reduced to nine hours, eight hours, you know, they want to, you know, increase the clock. They might start going, you know, To relate back to what we're talking about. Sensitive data, start saying, look, we know we can see you trying to restore, but guess what? We've already got all your customers, you know, passport details,

Jon:

That's why, that's why I think that's why I think anyone who might take that call should have some sort of specific resilience training. They should, you should definitely get someone on a room, you know, on a teams call, get someone in to do the, to play that other site. Cuz I think that just going there just sort of half experiencing that pressure, I think would be a very good, um, you know, just to try and keep your, keep your head, you know, um,

Rob:

but there's also the fact like they might, they may branch out to other groups of communications. They might start bringing other stakeholders in. They might just go to all company like to start putting a bit more panic. You know, again, if you are lower down the change employee, you worried about your, your job and you start putting the pressure on you start, you might ring your local newspaper. So you start bringing out like there's, there's lots of different ways that can apply that pressure, um, to try and ramp up as say, ramp up a successful payment.

Jon:

Okay. So, um, so we're in there. Um, we we've, we, so are we gonna do a branch now where we've paid and where we haven't paid? Should we do that as a, as a what the consequences might be? Or do we want to get more into prevention? We're gonna do prevention at some point. Aren't we, I know, I'm not saying anything's a hundred percent, but the sort of things we can do to

Rob:

yeah, we'll get, we'll get, we'll put a bit of light at the end of the tunnel after we've, uh,

Jon:

Ah,

Rob:

gone through all this stress

Jon:

I'll be able to sleep tonight. Hopefully. Yeah.

Rob:

um, well, I mean, yeah, in terms of, in terms of impact the business, it's, it's gonna vary between organization, organization, but the, the key thing, and again, we said at the top of the top of the discussion was that that ransome payment is very unlikely, gonna be your biggest impact and that if you make the payment, um, so as I say, if you do get on the route of making the payment, which normally happens either, I say you do it straight away as a reaction. or you haven't been able to understand the blast radius, you haven't been able to analyze the, the scope of the threat. You haven't been able to retrieve, you know, viable backups or the recovery time is so long that it's not actually justified. Those will lead you to pay in the ransom.

Jon:

I mean, Rob, if you were really sophisticated, you would deliver to the CIO the business case, um, because you do all your prep and you would just show the ransom that you need to pay. Plus the, versus the damages that they're gonna be occurring, you know, you mean, if you really wanted to kind of lay it on thick, that's what you would say, um, you know, to make that decision really quickly. But that's what you're saying, isn't it there'll be a point where you'll realize the way they've engineered. It is the, the ransomware might be rounding era compared to what the consequences are. Um, it's just happens that that ransomware is significant funds for the criminals. You know, there's a, there's a, there's a point where it might even become transactional. It all becomes like digital protection money where every month they come around for the, you know, for the protection. I mean, I've just, I'm just going

Rob:

kind of. Yeah, no,

Jon:

yeah. Digital, digital mobsters.

Rob:

thing you can be, the thing you can fall victim of which to be fair we're doing right now is you get tunnel vision of the immediate impact. So every cost we mentioned there is probably a, a year one cost of a ransomware attack. As you say, it's that immediate downtime, it's the need to recover. It's the rent and payment. It's the, you know, the, the audits, that thing you're then gonna have a further sort of two years probably of. Getting ready for, for legal issues gonna need consultancy. You're gonna probably review all your security procedures, infrastructure. There's gonna be investments there. You're gonna have sort of, um, you know, brand image

Jon:

the cyber insurance, that, that, that's also consideration isn't for year two or beyond

Rob:

Yeah. So say you, you might get lucky and have cyber insurance help you with year one costs. And I say, you might get lucky if anyone's ever had any dealing with an insurance company in any capacity.

Jon:

Yeah.

Rob:

They're not, they're not, they're not too forthcoming with the

Jon:

isn't designed to pay out loads of money. Yeah.

Rob:

No. Um,

Jon:

you've got legal, uh, you've got the security review, legal implications, GDPR, you know, there could be all sorts of, um, damage, potential damage that you've, you've, you've inadvertently caused, you know, because the data's that you are, that you are looking after on behalf of someone else you've, you've given out cyber insurance, you mentioned brand.

Rob:

So you got the, you got the brand impact. What's that gonna have on your, your revenue streams? How are you gonna be able to recover?

Jon:

All these areas that you're talking about is like a, kind of like a, you know, what happens to a company when they have a major PR disaster, you know, that's, it's, it's, it's up there. It could be like, you know, when, if there's a recall of a product, you know, that's dangerous or something like it's right up there with the, you know, these are, these are very, very big events that can impact. The the, the total value of the company

Rob:

It's yeah, it's precise that and as well, you've gotta remember that you are, you know, as, uh, as members of the public, we're becoming more and more educated in the importance of our, our data and where it's used and, and all these things is getting more and more people are responding.

Jon:

if you are. Um, if you are in the professional services space, Rob, uh, or you're doing public sector tendering or any tendering process, um, there's normally a stock question in a tender that says, you know, are you in litigation with any existing customers? You know, those sorts of things I can imagine in tender, this might be common. I there's been a couple of years since I've done attender, but, um, it might say, have you ever, you know, has your business ever been subject to ransomware and have you paid, do you know what I mean? There could be. I mean, that could be a very interesting piece of due diligence. Couldn't it? In the M&A space.

Rob:

Yeah. And it would be interesting. Cause I guess, how would you, how would you really legislate that? So say if you've already avoided letting anyone know that payment catch on the radar, like where does that fall in terms of making those declarations again, you.

Jon:

Yeah, I'm, I'm pretty certain, it'd be, it'd be material, you know, it's a, so you've got to reveal it. If it's something that's, uh, especially if you're saying 60% get, you know, get approached again. Um, you know, there is a potential there isn't there.

Rob:

Yeah, a hundred percent. Um, and yeah, to say we've only got sort of years, years, one and two there say you've normally go on for another 3, 4, 5, 6, 7. In terms of rebuilding reputation, rebuilding business activities say continued investment. And a lot of these costs as well, we're talking about aren't necessarily like, you know, the fallout, it's not necessarily getting sued, having to pay big legal costs. Although is a big part of it. It's having to pay the advisors, the consultants, the lawyers, the external people come in and do full and that kind of thing. And it just ramps up and up and up. And it feels like a problem that you can't get away from. When, as I say, when you talk about the problem in your head, you only picture that first

Jon:

yes. yes.

Rob:

twenty days maybe. but it's something you're gonna have to wear on your back for, for years and years

Jon:

you're absolutely right. That's like I said, at the beginning, remember, it's the sort of, you know, the ransomware letter you pay or whatever person gets returned thankfully, and everyone's happy and that's the end of it. This is much more incipid, isn't it? It's like that super tanker, uh, it's got all this momentum that you just can't stop, you know, when you encounter it, it's too late because it's just too, too powerful to stop. Um, okay. Um, how, how close are we robbed to light at the end of the tunnel? You know, are we, are we still, we still, and the there's that, uh, saying the darkest hour is the hour before Dawn. Um, which I love, but recently I think I heard it on the telly. Someone said the darkest hour is the hour before total oblivion, which was a slightly different it's a little bit glass, half empty. I'm more of the hour before Dawn sort of person. But, um, it is, uh, yeah, it's the strange thoughts that, that, that are going through my mind. Roberts, I'm trying to sort of immersing myself in the scenario.

Rob:

It, it it's tricky. And so much of it is how you a, how you prepared for it and how you respond to it. Um, it, it is a very, a business defining and it's a career defining event. So you tend to find, um, there tend to be large events of attrition. A lot of people, uh, will either make or break their careers based on how they respond to that. A lot of people just get burned out from the stress of dealing with it and have to change jobs or whatnot. So on an individual basis, there's,

Jon:

And, and just, if anyone hasn't been there for burnout, um, we're talking about people who, I mean, I heard some anecdotes. Okay. So I'm not giving any secrets away. This was just anecdotes I'd heard, but in some scenarios, folks were, hadn't gone home for like three weeks. They were sleeping under desks. Uh, because they were just doing absolutely everything they possibly could to try and prevent or mitigate or, you know, beat the attack. Uh, but as the days and weeks went on, there's just, you know, they couldn't stop it. So I mean that, that's, that's really, that's gonna really affect you badly because you're, you're, you're a professional, you're an engineer, you're doing everything you think you can do, but you are, but you are not, you are doing the two steps, forwards, three steps backwards and it's, and it's, you know, I think that's, we should have a lot of respect for people who've been through the early ones. Um, you know, and actually, you know, just to shout out to those folks, um, once if they do recover and they're ready for getting into operational life again, albeit maybe in a different company, you know, these are people who've really been there and done it. So they're actually extremely. Valuable folks to have on the team because they they've lived it. Um, and I, you know, my hope is that some of those folks will probably end up being very successful, you know, potentially Rob in companies like yours and other companies, because they can start to advise on the pain points because they've actually, there's nothing like having been through one, I would imagine.

Rob:

No completely, you become invaluable in that, in that fight against the attacks. And I think the conversations, when you speak to people who have been in the heart of an instant versus people who are just aware of the threat, way people discuss it, the way people react is is, is so different. And. Yeah, just echo, like your heart goes out to, people have to deal with that. And remember, these are people who, you know, we go, we made that joke about CIOs, not signing up to be a hostage negotiator, but equally, you know, this level two security analyst did not sign up to carry the weight of the business on their shoulders for, you know, four sleepless nights or three weeks or however long the it goes on for. And it's, it's a huge amount of pressure because the, the fallout of these attacks have, have impact on society. That's the, that's the

Jon:

Yeah. And, and, and, you know, pressure, I mean, you know, we, we, we know we're all under this enormous amount of, of pressure in this, in the world that we, that we currently live in, uh, and all the things that we've been through. Um, yeah. And folks, like you say, they're not mentally prepared, uh, for it. And, and actually some of them shouldn't be like you say, they should just should not be put in that position in the first place, but they were kind of like the, um, I dunno if anyone's gonna remember this phrase, but there's the phrase of the Bobby on the beat. So they were the, they were the single policeman that arrived at the scene of the huge accident. Who's just basically waiting for the serious incident rep, you know, everyone to turn up, but they've gotta hold it together for that initial, um, moment, uh, or moments and keep, keep, keep as calm as possible. So, yeah. Um, yeah, uh, you know, cudos to them.

Rob:

Absolutely. Absolutely. Um, but yeah, I guess let's talk about how to avoid them, how to, how to not let it happen, how to not have a horrible time in burnout and, and pay ransoms and, and all these, all this demo glue we've covered. Um,

Jon:

Yeah.

Rob:

and you know, there's, security's a, a huge landscape, so won't even attempt to, to tackle it. We'll talk very high level and I guess split it off into two key categories. You want to focus on probability. And you want to focus on impact, right? So probability is you just want to reduce the chances of a successful attack happening. And that's essentially with your, mainly with your perimeter security, right? That's building the walls around your data, around your organization, around your methods in order to keep these bad actors, the malware, these messages attacks out. So there's all the standard stuff, your email security, your, you know, your antivirus, you're threat hunting, your file walls, all these, these different areas. They're gonna build walls. And there's some really fantastic technologies out there now. And the, you know, the, the lengths we've gone to and, you know, the capabilities we've got now with, between machine learning, AI and these things to help with intelligent threat detection and not rely on, you know, things like signatures and, and whatnot is huge. Um, and actually the more you invest in that, the, the strongly defenses are there. The higher your walls will be. The problem you have is you build a high wall. They'll build a higher ladder, right? You're still a target. And you think of all the high profile examples that are out there.

Jon:

an arm's race.

Rob:

Yeah. You know, none of these, none of these big businesses that get hit by ransomware, I sat there without any, without firewalls, without any antivirus or basic things like that. Um, the, the, the second part of the probability, which is a huge piece, and it's again, get a lot more attention in recent years. It's the education of your, your employees and your users. Cause again, that's gonna be your biggest threat. That's the, that's the easiest entry point is an ignorant, a curious a, uh, greedy employee. Cause we all like being nosy. We like free stuff. We're not always concentrating too much. Um,

Jon:

This is the, this is the sort of very subtle background, social engineering. This is, this is getting in at the initial stages of, of, of, of infiltrating your systems pretty much through human activity, curiosity, um, not being alert. Uh, clicking on links. And like you say, free downloads. I mean, pretty much every firewall on the planet has a rule that says it stops free. Anything that's free, it just stops

Rob:

it.

Jon:

Um, but then, you know, if you've got, if you haven't got the right balance, um, in the culture of your business and you are, you know, people are like, look, I want to have access to this. I might need to do X, Y, Z. Um, that's where you've got to have a mature relationship with the business, um, about what you, what you allow to occur. But I think, um, the picture you've painted Rob, about the attack, that's the picture you've got to, you've got every board should go through this. And every, every MD, uh, CEO of, of an SME should go through this with his, his, or her top team. This scenario you've just described because, um, we'll get, we'll get a bit more into, you know, what, what you were talking about, but I just wanted to, to, to, to, to lay something over and what you just said, you talked about probability and

Rob:

mm-hmm

Jon:

So, what I'm imagining is, is for me to build the business case to fund, you know, this, this prevention, we've got the probability and impact, which, you know, working with yourself, we'd be able to fill in probability impact, et cetera, what I would do over the top of it. And forgive me, Rob, if you were gonna say this in any case, but over the top of that, I would overlay ability to control and I would do it in three ways. I would say what's our ability to control it right now. You know, you might just do 1, 2, 3, you know, do you wanna overengineer these things? Uh, a 1, 2, 3, 3 is no ability whatsoever. One is total control just to sort of an approximation. So what's our ability to control. Now, what is the, uh, is there technology available that would allow us to improve that? So literally, is there technology available or is it simply. A risk, you know? Um, and then, uh, the third one would be, um, uh, uh, you know, what's, what's the cost, uh, of that technology. So what you'd then have a do is you'd look at probability impact. You'd say, this is what we control at the moment. Uh, this is the total space we could control, and it would cost us this much, but it still leaves this uncovered and this, and typically this area that's uncovered is probably our human behavior. We can mitigate against it, but we can never, ever, uh, stop it. There's always gonna be different route. I mean, it's very simple, but maybe that, then you lay back to the board for the investment case and you say, look, this is what we can control now. And actually, um, what we can control now will mitigate this type of ransomware attack, but it won't mitigate all of them or, or, or actually we're still very vulnerable. You, you make an assessment and then you say, but however, with the investment that we've put, that we can put in place, we can change our, you know, our, our risk goes from here to here so that I don't wanna be

Rob:

no, no. I think I.

Jon:

but I'm trying to sort of build on top of your probability impacts really important, cuz obviously that makes you, you don't sweat those, you know, you don't look in the wrong area and you're focusing on the it's most likely to happen. It's gonna have a big impact, but it's that then ability to control

Rob:

Yeah, completely. It comes down to your comes down to your ability to execute in these areas. Um, and it comes down to your, your attitude towards the risk and the, what you deem to be the risk to your, to your business. So, yeah, your, your capability of how much is how much is gonna cost comes into it. Cause I say there's no point investing millions and millions and millions to try and save a, you know, a very small turnover business. Um, you know, I've heard examples before where it's been cheaper to rebuild a data center than it has been to recover an organization. It's, it's getting those, those different,

Jon:

but the other scary thing, when you say that, Rob, you know, we look at it and we say, you know, we've done this exercise and I'm making this number up, but we think it's gonna cost 15 million. Um, and then we know that the ransomware that we're, that we're most likely to get is for half a million. I hate to say it, but this starts to become a, pay the ransomware in the business case as an option line. I mean, I'm not saying it should be, but,

Rob:

no, no, no. You're right. But I

Jon:

some people that are money minded would look at that straightaway and go, well, hang on a minute, Jon, if we did that, that's effect at current rates, that's 15 years worth of ransomware, you know, aren't we better just paying I'm I'm playing devil's

Rob:

no, no,

Jon:

but do you, do you see

Rob:

it's completely, it's completely right. It actually, it's a really good point cuz it relates into then where that approach is mitigating. The impact is so important. So we covered a lot about the probability about keeping people out, which is where most people jump to. First of all, how do we stop these things getting in you're right. It comes down to a point. You've gotta have a balance between. What's worth the ransom versus the downtime. So when you look at the impact, actually the conversation there is how quickly can my business recover. So if you focus time and attention on having a safe inpenetrable set of backups, plus the ability to accelerate your instant response, to know what you're dealing with and accelerate that recovery as a result, that's, what's gonna lower the downtime cost versus that Ranson payment. And that's how you're gonna beat the ran merit attacks. So you put up all your probability stuff from coming in. So you only gonna have to deal with, you know, a smaller number of, of attacks, but worse off the basis that, you know, if people didn't see this coming already, that you've really gotta have the attitude of not if, but when, if we can lower that impact by mitigating all the facts, we went through, making it easy for teams to react to things, make it easy for teams to understand the risk and making it easy for the business to recover. Then suddenly those random payments aren't as scary.

Jon:

I, I get you, I'm taking notes by the way, because, um, I wanted to, um, you know, this, I think we said it earlier as well. When we said don't ask the question, have you backed up? You asked the question, can you recover? And you've just, you've just double underlined that. So your mentality needs to be in the how, how fast can we recover and making sure that that where you're recovering from is safe. Um, but uh, I wanted to run past you, you know, in, in a modern business now. I mean, there's no such thing as a single modern business, that's such a stupid generalization, but in, in businesses I've worked in the amount of data. That's in play, uh, can be measured in petabytes. Um, and I, I first encountered a petabyte in 2004, I think it was, um, and it was to do with, uh, a lossless a storage of images, um, absolutely ginormous. And, um, they had a presentation, they were trying to explain how big a petabyte was and the best we could come up with. And there might be someone who might comment on this being inaccurate, but the best that they came up with was if you take a single galaxy, the number of stars that exist within a galaxy is kind of how you kind of count up to a petabyte. So imagining it as they're absolutely ginormous. So you can measure normally the number of petabytes in a decent sized company will be measured, you know, maybe in less than 10 or something like that, but it's still massive. So. Technology, Rob, is it there that you can still recover quickly, but scale and size and you can deal. I mean, I know zetabytes have been discussed by, you know, petabytes would be folks, uh, from certain industries, which would laugh at petabytes, but it is enormous, massive amounts of, of data. Is that a problem

Rob:

so naturally does it, does, it does post some challenges, but the, the important thing is not to focus on. Mass recovery. Um, I mean you can, you know, the technology is there to, to recover at mass very quickly, but actually the, the key thing, when it comes to business, continuity is identifying what your critical applications of critical workloads are

Jon:

Yeah. It's your working process? Isn't

Rob:

and get yeah. And getting them up. Not only getting them up quickly, but getting them in the right order, that sounds silly, but you need to orchestrate it to make sure that you're bringing up the right apps, the right order so that they work together. So you can get your core business functions

Jon:

No, no, I, I get it. But you know, some businesses have to keep data for regulatory reasons. Um, and so that, uh, others keep, uh, data for that plus the just in

Rob:

yeah,

Jon:

Um, so, so when you are running the business that this restore strategy needs to be restoring, the stuff that the business is running on this month. Yeah. You don't wanna be bringing back stuff that you have, you know, you may need to use from an archive six years ago. So this is, this is like you're saying your you've gotta be very, very pointed about your recovery strategy,

Rob:

Yeah, absolutely. we, we

Jon:

to, to prioritize it, to get the company up and

Rob:

it's to make it's it's it's, as I say, it comes back to that point of it's recovery in a business continuity situation. That that's the important thing we're not looking to bring back. As you say archives from 12 years ago that we've got for legislation reasons that no one is ever gonna look at, it's not about just right. We've been hit. Let's just bring everything back to make sure it's safe. It's going right. What does our business need to run? What does it need to, you know, how do we stop the revenue being implicated, how to keep the shareholders happy? Let's get, let's get it back up and running because

Jon:

could you put something into your data? Could you put a signature into your data so that when it was shared in the dark web or wherever it surfaced, there would be a fingerprint on it to show that it had come through an unofficial means,

Rob:

sort of like a, is it smart? Is it smart water that they use to like tag

Jon:

yeah, I thought I'd just invented. I thought I'd just invented that live on this call, Rob, but, um, and if it is a thing that nobody wants to talk about, we don't have to, but do you see what I mean? Is there something, is there something we can do to kind of infect the ransoms so that, um, they might not, you might not be able to trace them through the pattern of the transaction because of Bitcoin and the ledger, and everything's just impossible, but maybe you can do it at. Uh, anyway, I'm not, I'm not suggesting Rob. We, we brainstorm that one right now, but

Rob:

It's no, it's a really

Jon:

just, it was just a

Rob:

it's a really good, good thought. And there's probably a few people listen, absolutely screaming about how this would be done or how you can't do it or whatnot. I mean, I, I don't actually know myself, but what I would

Jon:

but I think I like the, uh, yeah,

Rob:

what I would say is actually that for the, for the gangs, you know, they're already sophisticated criminals. They're already spinning up and spinning down very quickly, moving around stuff and being tracked at the end of the day, your data's gone. The, you can, you can tag

Jon:

yeah. I don't think, yeah, no, I don't think the smart water. Yeah, yeah, absolutely. No, I, I totally get it. So I wa I wasn't, uh, your trademarking, um, smart water were smart, smart data, water, whatever we're gonna call it. Um, where I was coming from was more of a strategic countermeasure to the, to

Rob:

the wider problem. Yeah.

Jon:

the, to, to, to the 20 billion a year market, because that's not gonna stop you. But if it was something that everyone signed up to, you know, it might, it might start to create a, a bit of a blueprint, but any road just, um, inventing something, it doesn't exist. Um, so, so, so let's, let's, let's get grounded again. Sorry, Rob, for going down all these rabbit holes. Um, so yeah, light light at the end of the tunnel. So we're focusing on, uh, being able to recover, uh, quickly, but also making sure that the, the, where we're, where we're recovering from is absolutely squeaky, clean, lovely, protected, not infected. Just, you know,

Rob:

Yeah.

Jon:

how on earth do you do

Rob:

Don't don't just focus on, you know, the technically of how quickly I can get my data back, or, you know, how good my lines, all these kind of things. You, the, the key point to the recovery strategy is the, that instant response part that I mentioned earlier, you need to be able to assess the data, assess the problem, know exactly where that, that runs and where is infected. So you can isolate those files and not, not bring them back online to infect the others. Um, you need to have an idea of, you know, where we got any to have an idea of what data's been compromised. You can write with that as well. It's that whole analysis and investigation piece before you then bring up what you need to spin back up to bring your business back online. It's, it's finding the equilibrium between the, between the two it's the key route to recovery.

Jon:

cool.

Rob:

Um, and yeah, I guess the, the final thing, which, again, isn't the final thing, cause we've mentioned it so many times, but it. It's all about that preparation. It's all about that run throughs. It's all about asking the difficult questions. It's all about running through every stage of the business. Cause if you're preparing times of peace, when war happens, you'll be, you'll be ready to deal with it. And that's the key thing is just treating it with the, the understanding and the respects that these incidents deserve, I guess, because they, the impact are catastrophic. So.

Jon:

And, and Rob your organization, um, you, you provide, um, I know you do lots of things, but one of the things I think you do is like a simulation, uh, or you, or you run events. I dunno if you do it specifically for like an individual customer or if you have events where you invite people in, but don't you guys do, you know, the run the scenario. So folks are kind of going through, uh, the process. You're trying to basically sort of do a version of, of this podcast, but, but, but with folks around a table and phones ringing, and

Rob:

it's spot on. So we do, we run, um, live interactive role plays of, of what it's like to be in a visitor ransomware attack. When people hear that, I think they, they immediately cringe a bit and get a bit, you know, if the standard British response of being like, no chance am I doing that? But, um, I've, I mean, I've done it, I've done it three times myself and each time I've learnt something new from it. And it's, um, a really valuable way of, of living, living that ransomware, seeing the CIO's response, seeing the CEOs, response, seeing other businesses, business teams

Jon:

do you know, um, having spoken to you tonight about this, it makes me want to do, to go through that process. But before speaking to you, there's a nervousness about being a senior professional engineer and somehow not knowing the answer makes me, you know, you know, there, there's a little bit of that kind of, um, stupid hubris or professional pride that gets in the way of saying actually we don't we've I haven't encountered it. He says touch wood. Um, but I do know from my major instant management days, that rehearsal is such an important part of, of preparation scenario, planning and rehearsals. Um, And I think, you know, the triple, uh, jeopardy that you described, which I think we got to quadruple, or even whatever, whatever the one is after that. Um, but those sorts of things that's, that is the sort of stuff that really will come out, um, during a, so it needs to be a pretty safe environment. Yeah. It's no one's getting judged there. Isn't like a report going back

Rob:

no it's

Jon:

I'm sorry, Jon, Jon, Jon was fine after he stopped crying. Um, or you know, that, you know, that sort of, that sort of thing,

Rob:

no, like, I mean, I think like anything that, that actually brings change and, and, you know, and helps people move forward. You, you need to leave your pride at the door. You need to go with an open mind and it is a safe space for everyone to interact and understand and have these. Difficult conversations, um, that you wouldn't necessarily typically have. And again, have yourself challenged because if you run through the simulation yourself, no, one's there to pick up on, on your flaws. Um, and it's just a, I think it's a really eye-opening opportunity to, to do that. And, um, yeah, we do. We, we, we run them publicly or say if, if people do wanna do them for an individualist, the company, we're more than happy to come in and, and run, run through them.

Jon:

So, so Rob, will it be all right if in the description, um, we on the podcast, uh, sorry, on the YouTube channel, we can put in the description. Uh, uh, contact details or something if, uh, folks wanted to reach out to you for, you know, to, to, to talk about that. Um, what do you wanna say to the podcast listeners? Because there are folks who just listened to the podcast and don't go to YouTube. You know, it's not, YouTube's not for everyone, a lot of people, um, I I've heard someone I think did three of three, three episodes on a car journey, you know, so it's sort of listening in. So, uh, I was thinking, wow, that's, uh, I was trying to work out where, where they were driving to for, for three episodes. But, um, how would they, how would they reach out to you, Rob? Um,

Rob:

Yeah. AB absolutely. So, you know, we, um, we've run through a lot of fairly as we, as we said, fairly difficult topic today. So there's fairly sort of stressful things to think about. And if it's something you wanna learn about a little bit more, um, where I work and where, where we partner with companies, where we help them is, is really about that impact piece that I spoke about at the end. It's all about mitigating the, the impact of a successful attack and helping businesses to recover. So what I would say is, if you do want to learn a little bit more, have a have a conversation, um, probably the easiest way is to, is to reach out over LinkedIn, um, and, and contact me. So we, um, my name's. Rob Eadie and I'll pop up smiling. Um,

Jon:

Yep.

Rob:

me because there's some, uh, there's some horrendous articles about a criminal in Scotland. Who's done some pretty horrible things. So don't Google it. Put me into LinkedIn. It's a, a safer space.

Jon:

Well, actually, to be honest with you, Rob, you do want people to Google because the more they Google you you'll

Rob:

beat the algorithms.

Jon:

higher. Literally you'll and then, and then that, that, that other Rob will go below the fold as they say, and, and, you know, you'll be cool. So, uh, it'll just have to, we'll have to go through that. At the end of the day we're engineers and there is a technology stack and there's probably a few technology stacks to think about, you know, the whole point you were making about immutability saying, well, actually, Jon, it's not that new think of a tape CD rom that sort of thing. So if equally, if someone wanted to reach out to talk to you about the technology stack that goes around that focusing on recovery and making sure your backups are lovely, pristine, safe, you know, that, that, that technology element, is that also something that people can reach out to you for via LinkedIn?

Rob:

Yes. Yes, absolutely. Um, as I say, we, we tend to have the conversation around the matter first, and then we don't leave you hanging in the dark. We have, we have technologies that will help you mitigate that impact. So, um, so yeah, so whether it's just one of you just wanna have a chat about it a little bit more, or as you say, if you're interested in any sort of the, the live experiences or role plays we can help with, or if you wanna talk about the technology stack, then I'd say, yeah, reach out more than happy to set up any meetings, calls, discussions, whatever. Um, we'd really evaluate it to

Jon:

Well a massive thanks Rob. For, I know it's taken a while, mainly on my part, you know, I've had a few interruptions, uh, unfortunately, cause we were gonna do this, I think quite a bit early, like two months ago. I

Rob:

been a bit of time in the making, but it, no, it it's been great. I really, uh, really appreciate having it. It's been a, it's been very enjoyable.

Jon:

I've really enjoyed it as well. And um, I just wanna say to you, it's been a little bit torturous watching you quoff a few sips of beer. I'm guessing that was beer. Uh, here's me. I'm dutifully waiting until we've finished our session. I've got, I've got a couple of cold ones in the fridge, uh, ready to have on a Friday evening, but seriously, Rob, uh, massive, thanks to you.

Malcom:

Remember, prevention is better than cure, there are lots of ways you can tip the balance in favour of not paying by improving your perimeter and critically focussing on how quickly you can recover and contain. subscribe to the channel now to get access to a catalogue of business technology topics, that are easy to digest and share. Click the icon of Jon's face, such a handsome fellow. Honestly, some of the things they programme me to say CTIO 1 O 1 Business Technology Simplified and Shared. Subscribe now. Sponsored by Fairmont Recruitment, Hiring Technology Professionals Across the UK Europe