We need to design secure control access
All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points. This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly. Here's hoping we don't make it worse, --Bart
What\when was the most recent beach? The hypervisors are accessible publicly? Why no VPN/VPC. I've been in admin/networking/devops world since 2000 and currently attending to get my BS in CIS/Cyber Security... so if nothing more, I'd like to tag along and learn more from this real world scenario from I'm sure way more experienced users. On Wed, Feb 8, 2023, 3:34 AM Bart Kus <me@bartk.us> wrote:
All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points.
This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly.
Here's hoping we don't make it worse,
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Your background sounds like you'd make meaningful contributions, so I'd encourage you to consider participating in read-write mode, not just read-only. We got hit by this a few days ago on several HVs: https://www.bleepingcomputer.com/news/security/massive-esxiargs-ransomware-a... I'll avoid getting into the technical weeds question, to keep this thread focused on working group formation. --Bart On 2/8/2023 3:55 AM, Jamie Owens wrote:
What\when was the most recent beach?
The hypervisors are accessible publicly? Why no VPN/VPC.
I've been in admin/networking/devops world since 2000 and currently attending to get my BS in CIS/Cyber Security... so if nothing more, I'd like to tag along and learn more from this real world scenario from I'm sure way more experienced users.
On Wed, Feb 8, 2023, 3:34 AM Bart Kus <me@bartk.us> wrote:
All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points.
This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly.
Here's hoping we don't make it worse,
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Bart, Have you guys tried to get the decryption keys for esxiargs ? I work in cyber security and it was announced that CISA had released the keys to help decrypt folks impacted by the ransomware attacks https://www.bleepingcomputer.com/news/security/cisa-releases-recovery-script... 73 Wade W7ITL On Wed, Feb 8, 2023 at 4:09 AM Bart Kus <me@bartk.us> wrote:
Your background sounds like you'd make meaningful contributions, so I'd encourage you to consider participating in read-write mode, not just read-only.
We got hit by this a few days ago on several HVs:
https://www.bleepingcomputer.com/news/security/massive-esxiargs-ransomware-a...
I'll avoid getting into the technical weeds question, to keep this thread focused on working group formation.
--Bart
On 2/8/2023 3:55 AM, Jamie Owens wrote:
What\when was the most recent beach?
The hypervisors are accessible publicly? Why no VPN/VPC.
I've been in admin/networking/devops world since 2000 and currently attending to get my BS in CIS/Cyber Security... so if nothing more, I'd like to tag along and learn more from this real world scenario from I'm sure way more experienced users.
On Wed, Feb 8, 2023, 3:34 AM Bart Kus <me@bartk.us> wrote:
All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points.
This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly.
Here's hoping we don't make it worse,
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing listPSDR@hamwan.orghttp://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Nice! No, didn't see this yet. We have a copy of the file systems though, so hopefully can apply recovery keys there. Thanks muchly, --Bart On 2/8/2023 7:45 AM, Wade W7ITL wrote:
Bart,
Have you guys tried to get the decryption keys for esxiargs ? I work in cyber security and it was announced that CISA had released the keys to help decrypt folks impacted by the ransomware attacks
https://www.bleepingcomputer.com/news/security/cisa-releases-recovery-script...
73
Wade W7ITL
On Wed, Feb 8, 2023 at 4:09 AM Bart Kus <me@bartk.us> wrote:
Your background sounds like you'd make meaningful contributions, so I'd encourage you to consider participating in read-write mode, not just read-only.
We got hit by this a few days ago on several HVs:
https://www.bleepingcomputer.com/news/security/massive-esxiargs-ransomware-a...
I'll avoid getting into the technical weeds question, to keep this thread focused on working group formation.
--Bart
On 2/8/2023 3:55 AM, Jamie Owens wrote:
What\when was the most recent beach?
The hypervisors are accessible publicly? Why no VPN/VPC.
I've been in admin/networking/devops world since 2000 and currently attending to get my BS in CIS/Cyber Security... so if nothing more, I'd like to tag along and learn more from this real world scenario from I'm sure way more experienced users.
On Wed, Feb 8, 2023, 3:34 AM Bart Kus <me@bartk.us> wrote:
All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points.
This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly.
Here's hoping we don't make it worse,
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Hear, hear, Bart! As an infosec pro, I was a bit appalled after first installing HamWAN and seeing such lax security, akin to leaving the front door open all day&nite of your house in Sodo. I removed the remote access and reporting configuration from my client nodes for this reason, but now I hear the control nodes have their doors open? Recipe for disaster and subsequent need for DR that can be prevented. Stephen W9SK On February 8, 2023 3:34:17 AM Bart Kus <me@bartk.us> wrote:
All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points.
This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly.
Here's hoping we don't make it worse,
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
As I am retiring from the Goog on Friday, I will have more time to get involved in this project. Count me in. -Doug- On Wed, Feb 8, 2023 at 8:09 AM Stephen Kangas <stephen@kangas.com> wrote:
Hear, hear, Bart! As an infosec pro, I was a bit appalled after first installing HamWAN and seeing such lax security, akin to leaving the front door open all day&nite of your house in Sodo. I removed the remote access and reporting configuration from my client nodes for this reason, but now I hear the control nodes have their doors open? Recipe for disaster and subsequent need for DR that can be prevented.
Stephen W9SK
On February 8, 2023 3:34:17 AM Bart Kus <me@bartk.us> wrote:
All of the network's control points are on public non-firewalled IPs.
This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points.
This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly.
Here's hoping we don't make it worse,
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Ok, Bart, since there are now others putting their hands up to help, I'll do that same since the work load and hive brain will be spread around. In addition to having a MS in Cyber Security & Information Assurance, my previous BSIT was with Security Emphasis, and I have several security and network routing & switching certs, including CEH & Forensics Investigator, and if there's some Linux behind the curtain I can help CLI and security for that, also. I've also become somewhat MicroTik Router OS fluent since doing HamWAN, and have since used their products in other network installs. My consulting workload will distract me some, but I'm willing to volunteer some time to help out, just let me know if you need it. Tell me more about the loss of your source code, exactly how did that happen? Perhaps it can be recovered. Stephen W9SK -----Original Message----- From: PSDR <psdr-bounces@hamwan.org> On Behalf Of Bart Kus Sent: Wednesday, February 08, 2023 3:34 AM To: Puget Sound Data Ring <psdr@hamwan.org> Subject: [HamWAN PSDR] We need to design secure control access All of the network's control points are on public non-firewalled IPs. This is the worst security. It was done this way for the sake of simplicity. Our netops volunteers had to get up to speed with unfamiliar concepts like routing, funky netmasks, dynamic routing protocols, policy routing, VRRP, firewalls, MTUs, MSS control, IPsec, etc. We reaped the rewards of KISS from broader volunteer engagement, but lately we've been paying too heavy of a price for the awful security this simplicity creates. In the most recent breach we've lost important source code that will now need to be re-created. We escaped total disaster by the thinnest of margins, as one critical hypervisor just happened to be patched to 1 version higher than exploitable. This simplicity is not a good tradeoff anymore, so the time has come to introduce more complexity to the network to protect all control points. This is not a simple problem, since there are many fragility vs security tradeoffs, as well as complexity cost concerns. If you have experience or thoughts around this area, and can commit to a few weeks of design and implementation work on this project, please indicate your interest. We'll assemble a small working group in the next few days and start discussions. I expect the working format will involve some virtual meetings, since email is not high bandwidth enough to hash out everything quickly. Here's hoping we don't make it worse, --Bart _______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
participants (5)
-
Bart Kus -
Doug Kingston -
Jamie Owens -
Stephen Kangas -
Wade W7ITL