What have you done today thread?
- KungFooBob
- Posts: 13688
- Joined: Sat Mar 14, 2020 1:04 pm
- Location: The content of this post is not AI generated.
- Has thanked: 532 times
- Been thanked: 7217 times
Re: What have you done today thread?
Spoken with an ex-colleague.
The organisation has the Akira ransomware virus.
Rather than just encrypt the user data it seems to have totally destroyed the OS on all the hosts and deleted all the local backups. It's even borked the raid config on the backup server.
Not a problem it's all backed up by Veeam to the cloud...
Only they're encrypted backups and the saved encryption key seems to be incorrect.
They're utterly and totally fucked.
The organisation has the Akira ransomware virus.
Rather than just encrypt the user data it seems to have totally destroyed the OS on all the hosts and deleted all the local backups. It's even borked the raid config on the backup server.
Not a problem it's all backed up by Veeam to the cloud...
Only they're encrypted backups and the saved encryption key seems to be incorrect.
They're utterly and totally fucked.
- Yorick
- Posts: 16278
- Joined: Sat Mar 14, 2020 8:20 pm
- Location: Paradise
- Has thanked: 10113 times
- Been thanked: 6649 times
Re: What have you done today thread?
OopsKungFooBob wrote: ↑Thu Sep 28, 2023 11:31 am Spoken with an ex-colleague.
The organisation has the Akira ransomware virus.
Rather than just encrypt the user data it seems to have totally destroyed the OS on all the hosts and deleted all the local backups. It's even borked the raid config on the backup server.
Not a problem it's all backed up by Veeam to the cloud...
Only they're encrypted backups and the saved encryption key seems to be incorrect.
They're utterly and totally fucked.
- Yorick
- Posts: 16278
- Joined: Sat Mar 14, 2020 8:20 pm
- Location: Paradise
- Has thanked: 10113 times
- Been thanked: 6649 times
Re: What have you done today thread?
Bugger. It's only 23c. Not going in till it's high 20s.
It's 30c sun today, sill warm up soon
-
- Posts: 13479
- Joined: Mon Mar 16, 2020 7:57 pm
- Location: Milton Keynes
- Has thanked: 2609 times
- Been thanked: 6012 times
Re: What have you done today thread?
Bah! There'll be an admin somewhere who's got it all on a caddied hard drive in his desk "just in case".KungFooBob wrote: ↑Thu Sep 28, 2023 11:31 am Spoken with an ex-colleague.
The organisation has the Akira ransomware virus.
Rather than just encrypt the user data it seems to have totally destroyed the OS on all the hosts and deleted all the local backups. It's even borked the raid config on the backup server.
Not a problem it's all backed up by Veeam to the cloud...
Only they're encrypted backups and the saved encryption key seems to be incorrect.
They're utterly and totally fucked.
- KungFooBob
- Posts: 13688
- Joined: Sat Mar 14, 2020 1:04 pm
- Location: The content of this post is not AI generated.
- Has thanked: 532 times
- Been thanked: 7217 times
Re: What have you done today thread?
They only have one IT person. It was me. Then they sacked my replacement after 6 months and made the technical (product expert) guy who used to sit next to me IT Manager because he knew how to reset passwords. They're proper fucked if they can't find that encryption key (that I didn't even know existed, so I'm glad it didn't happen on my shift).Mr. Dazzle wrote: ↑Thu Sep 28, 2023 3:21 pmBah! There'll be an admin somewhere who's got it all on a caddied hard drive in his desk "just in case".KungFooBob wrote: ↑Thu Sep 28, 2023 11:31 am Spoken with an ex-colleague.
The organisation has the Akira ransomware virus.
Rather than just encrypt the user data it seems to have totally destroyed the OS on all the hosts and deleted all the local backups. It's even borked the raid config on the backup server.
Not a problem it's all backed up by Veeam to the cloud...
Only they're encrypted backups and the saved encryption key seems to be incorrect.
They're utterly and totally fucked.
- ZRX61
- Posts: 4847
- Joined: Tue Mar 17, 2020 4:05 pm
- Location: Solar Blight Valley
- Has thanked: 1446 times
- Been thanked: 1329 times
Re: What have you done today thread?
The correct response is "well, good luck with that, I'm off for a pint"KungFooBob wrote: ↑Thu Sep 28, 2023 3:25 pm They only have one IT person. It was me. Then they sacked my replacement after 6 months and made the technical (product expert) guy who used to sit next to me IT Manager because he knew how to reset passwords. They're proper fucked if they can't find that encryption key (that I didn't even know existed, so I'm glad it didn't happen on my shift).
- ZRX61
- Posts: 4847
- Joined: Tue Mar 17, 2020 4:05 pm
- Location: Solar Blight Valley
- Has thanked: 1446 times
- Been thanked: 1329 times
Re: What have you done today thread?
Double bugger. It ended up being $968,700,000.
The bad news: I didn't win
The good news: Neither did any other bastard... so $1,040,000,000 for Monday.
- Horse
- Posts: 11211
- Joined: Sun Mar 15, 2020 11:30 am
- Location: Always sunny southern England
- Has thanked: 5944 times
- Been thanked: 4926 times
Re: What have you done today thread?
Yesterday evening.
Another free event in the town. They're always a bit bonkers.
https://www.newburytoday.co.uk/news/don ... e-9332363/
The talented French company Compagnie L’Homme Debout will be gracing the town centre with Mo and the Red Ribbon.
The show tells the story of a child refugee named Mo and his optimistic outlook and fantastical journey towards a new life.
Another free event in the town. They're always a bit bonkers.
https://www.newburytoday.co.uk/news/don ... e-9332363/
The talented French company Compagnie L’Homme Debout will be gracing the town centre with Mo and the Red Ribbon.
The show tells the story of a child refugee named Mo and his optimistic outlook and fantastical journey towards a new life.
Even bland can be a type of character
- the_priest
- Posts: 1901
- Joined: Sun Mar 15, 2020 2:18 pm
- Location: Dwelling in Welling
- Has thanked: 1906 times
- Been thanked: 2164 times
Re: What have you done today thread?
Life changes for me.
It was announced today that The Bishop of Rochester is pleased to announce that The Revd aka the_priest is to be appointed as the next Priest -In-Charge of the Benefice of St Botolph’s Northfleet and St Marks Rosherville. We look forward to welcoming the_priest and his family to our Parish, hopefully before Christmas.
So big adventures ahead, some good biking roads not too far from there and so much to look forward to in life and faith.
It was announced today that The Bishop of Rochester is pleased to announce that The Revd aka the_priest is to be appointed as the next Priest -In-Charge of the Benefice of St Botolph’s Northfleet and St Marks Rosherville. We look forward to welcoming the_priest and his family to our Parish, hopefully before Christmas.
So big adventures ahead, some good biking roads not too far from there and so much to look forward to in life and faith.
Proverbs 17:9
One who forgives an affront fosters friendship, but one who dwells on disputes will alienate a friend.
One who forgives an affront fosters friendship, but one who dwells on disputes will alienate a friend.
- Yorick
- Posts: 16278
- Joined: Sat Mar 14, 2020 8:20 pm
- Location: Paradise
- Has thanked: 10113 times
- Been thanked: 6649 times
Re: What have you done today thread?
Any update?KungFooBob wrote: ↑Thu Sep 28, 2023 3:25 pmThey only have one IT person. It was me. Then they sacked my replacement after 6 months and made the technical (product expert) guy who used to sit next to me IT Manager because he knew how to reset passwords. They're proper fucked if they can't find that encryption key (that I didn't even know existed, so I'm glad it didn't happen on my shift).Mr. Dazzle wrote: ↑Thu Sep 28, 2023 3:21 pmBah! There'll be an admin somewhere who's got it all on a caddied hard drive in his desk "just in case".KungFooBob wrote: ↑Thu Sep 28, 2023 11:31 am Spoken with an ex-colleague.
The organisation has the Akira ransomware virus.
Rather than just encrypt the user data it seems to have totally destroyed the OS on all the hosts and deleted all the local backups. It's even borked the raid config on the backup server.
Not a problem it's all backed up by Veeam to the cloud...
Only they're encrypted backups and the saved encryption key seems to be incorrect.
They're utterly and totally fucked.
- KungFooBob
- Posts: 13688
- Joined: Sat Mar 14, 2020 1:04 pm
- Location: The content of this post is not AI generated.
- Has thanked: 532 times
- Been thanked: 7217 times
Re: What have you done today thread?
As of 6pm on Friday, they were still fucked.Yorick wrote: ↑Sun Oct 01, 2023 5:21 pmAny update?KungFooBob wrote: ↑Thu Sep 28, 2023 3:25 pmThey only have one IT person. It was me. Then they sacked my replacement after 6 months and made the technical (product expert) guy who used to sit next to me IT Manager because he knew how to reset passwords. They're proper fucked if they can't find that encryption key (that I didn't even know existed, so I'm glad it didn't happen on my shift).Mr. Dazzle wrote: ↑Thu Sep 28, 2023 3:21 pm
Bah! There'll be an admin somewhere who's got it all on a caddied hard drive in his desk "just in case".
All four European offices are down. All the data (15tb of it) is there in the cloud backup, they just can't decrypt it. I can't see the business being able to continue trading without the ERP (four separate instances of SAP Business One).
I think the biggest problem was the rush to flatten the existing server estate because they thought they could restore from the cloud. Restores had been tested, but using the on-prem Veeam Enterprise Manager that had the encryption key cached, wasn't until they tried to restore to fresh installed Hosts with the Enterprise manager not available (it was on-prem and wiped) they realised you needed the keys.
It's quite sobering how fucked a company is without it's data and how easily that data can be taken away.
With hindsight, they would have been better off paying the ransom.
- DefTrap
- Posts: 4336
- Joined: Tue Apr 14, 2020 8:23 am
- Has thanked: 2268 times
- Been thanked: 2086 times
Re: What have you done today thread?
I think paying the ransom is increasingly common. Companies have terrible disaster recovery policies and they don't give them adequate priority. You can't afford to be offline for weeks let alone lose all your data.KungFooBob wrote: ↑Sun Oct 01, 2023 5:37 pm
With hindsight, they would have been better off paying the ransom.
-
- Posts: 1882
- Joined: Sat May 08, 2021 5:14 pm
- Location: Another day without using algebra
- Has thanked: 66 times
- Been thanked: 1235 times
Re: What have you done today thread?
Have they tried turning it off, then back on again?KungFooBob wrote: ↑Sun Oct 01, 2023 5:37 pmAs of 6pm on Friday, they were still fucked.Yorick wrote: ↑Sun Oct 01, 2023 5:21 pmAny update?KungFooBob wrote: ↑Thu Sep 28, 2023 3:25 pm
They only have one IT person. It was me. Then they sacked my replacement after 6 months and made the technical (product expert) guy who used to sit next to me IT Manager because he knew how to reset passwords. They're proper fucked if they can't find that encryption key (that I didn't even know existed, so I'm glad it didn't happen on my shift).
All four European offices are down. All the data (15tb of it) is there in the cloud backup, they just can't decrypt it. I can't see the business being able to continue trading without the ERP (four separate instances of SAP Business One).
I think the biggest problem was the rush to flatten the existing server estate because they thought they could restore from the cloud. Restores had been tested, but using the on-prem Veeam Enterprise Manager that had the encryption key cached, wasn't until they tried to restore to fresh installed Hosts with the Enterprise manager not available (it was on-prem and wiped) they realised you needed the keys.
It's quite sobering how fucked a company is without it's data and how easily that data can be taken away.
With hindsight, they would have been better off paying the ransom.
Up at 7.50 today to watch a bit of water biking.
Then off to a friend who's electric pool cover was/has been a bit recalcitrant for the last couple of years.
Had to re-track 10 channels, then diagnose why it didn't work. For some reason it had 2 limit stop sensors, and of course 1 was fubar. So now waiting on a parcel delivery!
-
- Posts: 13479
- Joined: Mon Mar 16, 2020 7:57 pm
- Location: Milton Keynes
- Has thanked: 2609 times
- Been thanked: 6012 times
- DefTrap
- Posts: 4336
- Joined: Tue Apr 14, 2020 8:23 am
- Has thanked: 2268 times
- Been thanked: 2086 times
Re: What have you done today thread?
Yeah but this is the problem, total absence of giving a fvck, let's leave it to IT. Business Continuity is more than just IT getting some backups right.
- the_priest
- Posts: 1901
- Joined: Sun Mar 15, 2020 2:18 pm
- Location: Dwelling in Welling
- Has thanked: 1906 times
- Been thanked: 2164 times
Re: What have you done today thread?
I am so glad to have left the world of IT. God stuff is much better, you can pray about it!
We had full redundancy, online backups, off site storage, images and more. That was the one thing that we managed to drive into the CEO's head was that if the data was ever compromised, we'd have a GDPR nightmare on our hands, never mind the issues of lost data. Our lot trained as white hat hackers as well in being able to do penetration testing and to ensure the network was secured and backed up. That is all 5 years ago now and that landscape has probably changed again having moved fully into the cloud.
Sometimes it is just worth investing in things you might never need, because when you do need it, you look brilliantly clever and have saved the day! And keep good staff in place with decent pay and incentives!
We had full redundancy, online backups, off site storage, images and more. That was the one thing that we managed to drive into the CEO's head was that if the data was ever compromised, we'd have a GDPR nightmare on our hands, never mind the issues of lost data. Our lot trained as white hat hackers as well in being able to do penetration testing and to ensure the network was secured and backed up. That is all 5 years ago now and that landscape has probably changed again having moved fully into the cloud.
Sometimes it is just worth investing in things you might never need, because when you do need it, you look brilliantly clever and have saved the day! And keep good staff in place with decent pay and incentives!
Proverbs 17:9
One who forgives an affront fosters friendship, but one who dwells on disputes will alienate a friend.
One who forgives an affront fosters friendship, but one who dwells on disputes will alienate a friend.
- Skub
- Posts: 11860
- Joined: Mon Mar 16, 2020 5:32 pm
- Location: Norn Iron
- Has thanked: 9677 times
- Been thanked: 9700 times
Re: What have you done today thread?
I was out on the Zed today and called in with an old mate who's currently in a care home as his wife finds difficulty looking after him 24/7. He's younger than me,but badly disabled by a stroke about 7 years ago. I hate to see the way things have turned out for him,he's a clever bloke,but stuck in there with dementia sufferers and all the usual care home shenanigans. He's fallen twice already and is looking really bashed about.
He's always positive and cheerful and I get a massive reality check every time I speak to him. Sometimes we really don't know how fortunate we are.
He's always positive and cheerful and I get a massive reality check every time I speak to him. Sometimes we really don't know how fortunate we are.
"Be kind to past versions of yourself that didn't know what you know now."
Walt Whitman
https://soundcloud.com/skub1955
Walt Whitman
https://soundcloud.com/skub1955
-
- Posts: 1740
- Joined: Fri May 01, 2020 10:05 pm
- Has thanked: 233 times
- Been thanked: 403 times
Re: What have you done today thread?
I've been joking around the fact that some companies (and schools) put so much encryption and security processes in place that soon you wont be able to get into your own data/systems, it seems this has come true in your companies case.KungFooBob wrote: ↑Sun Oct 01, 2023 5:37 pmAs of 6pm on Friday, they were still fucked.Yorick wrote: ↑Sun Oct 01, 2023 5:21 pmAny update?KungFooBob wrote: ↑Thu Sep 28, 2023 3:25 pm
They only have one IT person. It was me. Then they sacked my replacement after 6 months and made the technical (product expert) guy who used to sit next to me IT Manager because he knew how to reset passwords. They're proper fucked if they can't find that encryption key (that I didn't even know existed, so I'm glad it didn't happen on my shift).
All four European offices are down. All the data (15tb of it) is there in the cloud backup, they just can't decrypt it. I can't see the business being able to continue trading without the ERP (four separate instances of SAP Business One).
I think the biggest problem was the rush to flatten the existing server estate because they thought they could restore from the cloud. Restores had been tested, but using the on-prem Veeam Enterprise Manager that had the encryption key cached, wasn't until they tried to restore to fresh installed Hosts with the Enterprise manager not available (it was on-prem and wiped) they realised you needed the keys.
It's quite sobering how fucked a company is without it's data and how easily that data can be taken away.
With hindsight, they would have been better off paying the ransom.
A MAT around my way (CSET) had a ransomware attack on a cluster of 28 schools, i think they ended up paying numerous hundreds of thousands of pounds to recover some of it. It also highlighted the fact they had no backups for any of their primary schools. The LA then forced the schools to come with them for support as the MAT's IT dept was forced to close. Ironically they didn't give them this as a choice as they said if they chose to go with a 3'rd party IT support company like the one i work for they wouldn't give them access to the data they had managed to recover, so in theory they ransomed the ransomed data back to the school!
Why wasn't someone monitoring the backups and doing a test recovery to ensure they are working at least once a month at your place?
- KungFooBob
- Posts: 13688
- Joined: Sat Mar 14, 2020 1:04 pm
- Location: The content of this post is not AI generated.
- Has thanked: 532 times
- Been thanked: 7217 times
Re: What have you done today thread?
I've been left 10 months. I did test restores when I was there.
The problem is the testing wasn't good enough. Yes they could restore data using the on-prem enterprise manager because it had the keys cached. I used it for full vms and individual files all the time.
Soon as the enterprise manager is out of the picture you have to use the key you made a note of when you configured the job.
Their noted key doesn't work. Might be a typo, might be the jobs were deleted and new ones created without the owner updating the notes.
Either way, it's not my problem, which I'm unbelievably pleased about.
The problem is the testing wasn't good enough. Yes they could restore data using the on-prem enterprise manager because it had the keys cached. I used it for full vms and individual files all the time.
Soon as the enterprise manager is out of the picture you have to use the key you made a note of when you configured the job.
Their noted key doesn't work. Might be a typo, might be the jobs were deleted and new ones created without the owner updating the notes.
Either way, it's not my problem, which I'm unbelievably pleased about.
-
- Posts: 1740
- Joined: Fri May 01, 2020 10:05 pm
- Has thanked: 233 times
- Been thanked: 403 times
Re: What have you done today thread?
I take it who ever created the key and has not got the correct key is in the shit then?KungFooBob wrote: ↑Tue Oct 03, 2023 9:36 am I've been left 10 months. I did test restores when I was there.
The problem is the testing wasn't good enough. Yes they could restore data using the on-prem enterprise manager because it had the keys cached. I used it for full vms and individual files all the time.
Soon as the enterprise manager is out of the picture you have to use the key you made a note of when you configured the job.
Their noted key doesn't work. Might be a typo, might be the jobs were deleted and new ones created without the owner updating the notes.
Either way, it's not my problem, which I'm unbelievably pleased about.
If that would of been me, i would of copied and pasted the key, took a picture of it, sent it to myself on email and saved it on about 3 different secure network drives to make sure i had it safe, too paranoid about those things myself after i had a bit locker issue with a missing or incorrectly written key about a year ago.