.

A cautionary tale for Penetration testers on live networks

<<

What90

Full Member
Full Member

Posts: 120

Joined: Sat Jun 09, 2007 2:23 am

Location: Syndey, Australia

Post Fri Mar 05, 2010 11:34 pm

A cautionary tale for Penetration testers on live networks

With many recent questions on “how do I exploit this?” or “if I do this on the network?” on the boards, this incident is worth thinking over before playing with a system or network that you don’t have written permission for.

I got a call from a friend, whom we’ll call him Dave, who was having some account lock out problems on his Windows domain. He was a bit spooked as the accounts were service accounts for backup, monitoring and SQL and they all locked out within minutes of each other. This stopped two major production systems in the middle of the working day and was causing him a stream of abusive phone calls on why “his” IT systems weren’t working and what the heck was going on.

Given Dave’s network was pretty well patched, they used a good account naming standards (so not accounts named backup, admin, etc) and every device had a decent AV on it, his admin senses were warning him something hoaky was going on. I directed him to look on his domain controller for the account lock out event id’s 539. Event ID 539 has the machine name, which means you can trace back the machine causing the lock outs. The machine name came back as XP, which was odd as the company machines are named after their asset numbers for accounting purposes. Pinging the machine’s name returned its IP address but no echo replies, so it appeared to be down. As I was asking questions, three more service accounts got locked out, all from workstation XP.

On a hunch, I asked him to check his DCHP server, look for the IP address there and tell me what the MAC address was. He found it quickly and noted that it looked different from the Dell machines MAC address details.
A quick look up on http://www.coffer.com/mac_find/ confirmed with was a VMware machine. Dave was somewhat disturbed by this as they are a windows only shop, including any virtualisation software. Dave’s LAN covers several floors and a large warehouse and has 16 switches, so we went with the old trick of searching the switches’ CAM tables for the MAC address to located which switch and port this IP address was connected to. As it happens the wiring diagrams were well maintained; the switch and offending port the MAC address had been connected was down one flight of stairs. Another three accounts locked out by the same machine name while we were doing this. A somewhat angry and confused Dave popped down stairs to see what was going on.

A now very excited Dave call back telling that a guy he’d never seen before was sitting at a spare desk, plugged in to the “evil” port, working on a shiny Mac laptop. He was on the way down to the warehouse to gather up some of the bigger lads to act as back up when he confronted the intruder.

Dave’s a big fan of the show 24, and imagination had got carried away with him. I told him to turn around, call his boss and check with reception if they knew this person was first. Reception proved to be more helpful than his boss, who had no idea. The “nice young man”, according to the receptionist, was a guest of the chief finance officer. Dave ran up the stairs to the top floor, burst in to the CFO’s office demanding to know who this guy was and why he was plugged in to Dave’s network.

The shocked then annoyed CFO stared at the wild-eyed, panting Dave and calmly replied he was a consultant doing some testing highlighted in the last audit. Dave, bravely attempted to not get himself fired on the spot, blurted out that consultant was the one who’d just stopped the inventory database critical to the warehouse operations. The CFO brain connected that this was something bad and, with Dave following, rushed down to the consultant’s desk and order him to cease all activity.

Dave had all the accounts unlocked and all the services running again within the next hour. He was asked to sit in a meeting with the CFO, his boss, the consultant and the consultant’s boss. The angry CFO was demanding answers on what had happened. The two from the consultancy exchanged looks and hand over the penetration testing contract and scope of work. All signed and agreed by the CFO.

To cut a long and embarrassing (for the CFO) story short, new auditors had recommended a penetration test be carried out to assess the internal systems as IT systems where now critical to the business. The CFO decided not to involve or inform anyone from IT, against the penetration testing company’s objections, as he always dealt with these types of financial matters. He ordered an extensive testing program including password attacks, all to be carried out on site and during office hours. Dave looked over the scope of work and was horrified what the CFO had chosen, the document clearly stated some of these tests carried risk, but had been signed off on.

The outcome from this was
1) The business  had a three hour outage in the middle of its working day when the consultant was in attacking mode by attempting password brute forcing on service accounts
2) The business estimated losing half a million dollars during those three hours
3) The two day pentest engagement was cancelled and a more sensible scope of work was drawn up
4) IT was included in on any and all audit requirements
5) The CFO got his wrist slapped.

Draw what you will from this tale, but always, always have written permission from someone in the business authorized to give it before even think about pen-testing a system.
<<

Ketchup

User avatar

Hero Member
Hero Member

Posts: 1021

Joined: Fri Jul 04, 2008 7:44 pm

Location: Philadelphia, PA

Post Fri Mar 05, 2010 11:46 pm

Re: A cautionary tale for Penetration testers on live networks

Chris, that's a great story.  Thanks for posting it, since there are many lessons to be learned there.  Coming from the consulting side of things, I was relieved to read that the pentesters had proper contracts in place. 
~~~~~~~~~~~~~~
Ketchup
<<

rattis

User avatar

Hero Member
Hero Member

Posts: 1172

Joined: Mon Jul 27, 2009 1:25 pm

Post Sat Mar 06, 2010 12:12 am

Re: A cautionary tale for Penetration testers on live networks

I was waiting to hear that they didn't have the right paperwork in place.

Good real life example of getting ducks in a row.
OSWP, Sec+
<<

What90

Full Member
Full Member

Posts: 120

Joined: Sat Jun 09, 2007 2:23 am

Location: Syndey, Australia

Post Sat Mar 06, 2010 12:38 am

Re: A cautionary tale for Penetration testers on live networks

According to Dave, the penetration testing guys were true professionals and had warned the CFO repeatedly, both verbally and written, on the risks he was taking. The CFO used the auditor recommendations as a have to do list.

As another display of  business awareness, the reason that only three accounts where being locked out at every five minutes was the consultant being as careful as he could. Rather than just running a brute force attack at the entire enumerated AD user names.

One really nice thing I liked, if Dave had appeared with his heavies, the consultant had a signed copy of the contract and all the contact numbers for the CFO with him at all times during the engagement.

Gold star to the Pentest team and unfortunately stock options for the CFO....
<<

UNIX

User avatar

Hero Member
Hero Member

Posts: 1244

Joined: Mon Apr 28, 2008 9:20 am

Post Sat Mar 06, 2010 2:27 am

Re: A cautionary tale for Penetration testers on live networks

Great write-up, thanks for sharing it with us. Personally I think it demonstrates some very important points which should be considered when performing any kind of pentest to a company. I am sure the CFO learned his lesson now, in quite an unpleasant but rememberable way though.
Last edited by UNIX on Sat Mar 06, 2010 4:02 pm, edited 1 time in total.
<<

apollo

Full Member
Full Member

Posts: 146

Joined: Fri Apr 04, 2008 7:44 pm

Post Sat Mar 06, 2010 3:38 am

Re: A cautionary tale for Penetration testers on live networks

Out of curiosity, did the pen testers recommend any strategic changes to your incident response procedures or any additional procedures to put in place in case this happens again ?  This is an excellent example of how having an incident response team with the proper professionals on it could have probably gotten things resolved faster.  You post this as a cautionary tale, with good reason, but it seems like there could have been some great positives come out of this that would last through a potential real attack.  Losing money is never good, but if you gotta lose money, make the most out of it that you can :)  I think if nothing else, some critical business points which are vulnerable to attack were exposed here.
CISSP, CSSLP, MCSE+Security, MCTS, CCSP, GPEN, GWAPT, GCWN, NOP, OSCP, Security+
<<

hayabusa

User avatar

Hero Member
Hero Member

Posts: 1662

Joined: Mon Jan 29, 2007 2:59 pm

Post Sat Mar 06, 2010 8:11 am

Re: A cautionary tale for Penetration testers on live networks

apollo has a great point here.  In my full-time gig (my non-security / non-pentesting gig,) the customers I support consider an outage as loss of $5 million every 15 minutes they are down, during any outage.  So time adds up quickly, and it's very important that they look at the overall results from the first test, analyze their response methodology and time to recovery, and make the most of their investment by modifying their incident response / recovery plans.  Too often, I see stupid outages, which take excessive amounts of time to recover, unnecessarily, solely because of poor planning, and in no way as a result of lack of training or knowledge.  And I've seen similar to this, when my customers (from said gig) have pentesting and analysis teams do work for them, too.  If it weren't for the fact that, in that job, I'm prohibited from engaging on that side of their activities, I'd just love to politely kick someone in the backside, for lack of a better term, and wake them up.  (BTW, doesn't mean that I don't drop subtle hints to said customers, under the radar...  ;) )

It's very important, as pentesters, that we analyze these types of situations and outcomes, and use them to promote positive, forward-thinking and proactive activities for our customers, so that in the end, they remain as safe as possible, remain stable, maintain uptime, and truly reap the benefits of our service.  A good pentest should not be limited to solely showing faults and holes, but rather, should encompass good business practices and promote cooperation, both with, and within, our customers, as well.

Thanks, What90, for a good reminder, both of how NOT TO do something, but also opening a good discussion of how TO do things better.
~ hayabusa ~ 

"All men can see these tactics whereby I conquer, but what none can see is the strategy out of which victory is evolved." - Sun Tzu, 'The Art of War'


OSCE, OSCP , GPEN, C|EH
<<

venom77

User avatar

Hero Member
Hero Member

Posts: 1905

Joined: Mon Dec 11, 2006 3:23 pm

Post Sat Mar 06, 2010 9:18 am

Re: A cautionary tale for Penetration testers on live networks

Wow, good story. Definitely a great heads up for everyone.

It reminds me of something an instructor had said in one of my courses about including some "out there" items in the documentation of what tests to conduct (e.g., following employees home, kidnapping, etc.) just to see if the person signing off on it actually reads it or just checks all tests and signs off. Then of course call said person up, ask whether they read through the paperwork and whether they really wanted to allow employees to be kidnapped. Crazy, sure, but certainly realistic if the information is important to someone.
<<

LSOChris

Post Sat Mar 06, 2010 8:42 pm

Re: A cautionary tale for Penetration testers on live networks

perhaps the takeaway from that experience should have been to not be able to lock out a service account that causes that amount of loss per hour. instead create a rule in their SEIM to alert on failed logins for those accounts as that should NEVER happen.
<<

former33t

Full Member
Full Member

Posts: 226

Joined: Sat Feb 14, 2009 12:33 am

Post Sat Mar 06, 2010 9:33 pm

Re: A cautionary tale for Penetration testers on live networks

Wow.  I don't even know where to start here...

Bill, I love the idea you shared about the outlandish ideas.  I'll be sure to use this both in consulting and at my normal job.  My boss routinely ok's things that I'm positive he doesn't comprehend (and as a result doesn't read).  A reality check might force a positive change.

ChrisG, you are right on the money about not locking out service accounts.  How you choose to deal with the problem of service account login failures is an administrative policy choice, but I have never seen locking them out have positive outcomes.  Instead of the one application using outdated credentials, everything that depends on the service account is down...

What90, I hadn't even appreciated the fact that the pentesters were taking the time to make sure they didn't lock out the whole domain in one fell swoop.  Good for them.  I'll make sure that I incorporate that into my next security assessment if I'm asked to attempt brute forcing of user accounts.
Certifications: CREA, MCSE: Security, CCNA, Security+, other junk
<<

LT72884

User avatar

Jr. Member
Jr. Member

Posts: 99

Joined: Thu Oct 15, 2009 3:11 pm

Location: Utah

Post Sat Mar 06, 2010 9:54 pm

Re: A cautionary tale for Penetration testers on live networks

DANG. We had a very similar experience on our network BUT the only diff was that it was not a pentest. some punk kid was trying to brute force our network because he was able to get the username file from our server using the null session with dumpsec tatic. Luckily we make sure our users have to follow our password policy. Still was annoying to get into work at 6 in the morning and have 33 accounts locked out.

Anyway. awesome post. Ill be sure to remember this for future use and recommendation.
<<

What90

Full Member
Full Member

Posts: 120

Joined: Sat Jun 09, 2007 2:23 am

Location: Syndey, Australia

Post Sun Mar 07, 2010 12:18 am

Re: A cautionary tale for Penetration testers on live networks

@Apollo

They got some great recommendations, but none of them transitioned off the report pages. Dave's company is a small place, around 250 staff, the IT "team" consists of Dave's boss (a non tech manager handed running IT as he showed some interest), Dave, and a junior desktop guy. Dave's incident handling process was to ring a friend (me, in this case) and ask for help.

I've seen the top ten objectives, management set, IT are to achieve this year and security wasn't one of them. In fact, security is only ever raised by the auditors who seem obsessed with IDS and what type AV they have, rather than if it's actually being used correctly :-) It's the normal excuse of we don't have the money to spend on security. The pentest was seen as a one off problem. Somewhat myopic...

Dave and the team have got a couple of positive outcomes from the event, but the overall postures is still reactionary for security.

@ChrisG
Account lock outs for all accounts after 5 failed attempts is on the auditors check list. It's a best practice apparently....

I pointed Dave to OSSIM http://www.alienvault.com/community.php?section=Home as a easy way to watch the network, but he isn't given the time to set it up. Instead he's currently been directed to look at Ms' SCOM product for better monitoring of his network.
<<

venom77

User avatar

Hero Member
Hero Member

Posts: 1905

Joined: Mon Dec 11, 2006 3:23 pm

Post Sun Mar 07, 2010 8:36 am

Re: A cautionary tale for Penetration testers on live networks

former33t wrote:Bill, I love the idea you shared about the outlandish ideas.  I'll be sure to use this both in consulting and at my normal job.  My boss routinely ok's things that I'm positive he doesn't comprehend (and as a result doesn't read).  A reality check might force a positive change.


Yeah, absolutely. Not only gets them to actually read stuff but also getting them thinking about the dangers that are really out there. The "bad guys" don't have any rules to play by.
<<

hayabusa

User avatar

Hero Member
Hero Member

Posts: 1662

Joined: Mon Jan 29, 2007 2:59 pm

Post Sun Mar 07, 2010 11:32 am

Re: A cautionary tale for Penetration testers on live networks

BillV wrote:Yeah, absolutely. Not only gets them to actually read stuff but also getting them thinking about the dangers that are really out there. The "bad guys" don't have any rules to play by.


Amen!  Absolutely the case.  The bad guys could care less about any rules of engagement, recovery plans, etc.  They're going to attack, regardless.  All of these things need to be taken into account, and should be clearly defined, and noted to the customer, so that they understand both risk and reward for testing and remediation.  This is why you have to put in the extra time, to show why pentesting services are valuable to customers, and to get mgmt to understand their own company's situation(s).
Last edited by hayabusa on Sun Mar 07, 2010 11:35 am, edited 1 time in total.
~ hayabusa ~ 

"All men can see these tactics whereby I conquer, but what none can see is the strategy out of which victory is evolved." - Sun Tzu, 'The Art of War'


OSCE, OSCP , GPEN, C|EH
<<

LT72884

User avatar

Jr. Member
Jr. Member

Posts: 99

Joined: Thu Oct 15, 2009 3:11 pm

Location: Utah

Post Sun Mar 07, 2010 3:53 pm

Re: A cautionary tale for Penetration testers on live networks

apollo wrote:Out of curiosity, did the pen testers recommend any strategic changes to your incident response procedures or any additional procedures to put in place in case this happens again ?  This is an excellent example of how having an incident response team with the proper professionals on it could have probably gotten things resolved faster.  You post this as a cautionary tale, with good reason, but it seems like there could have been some great positives come out of this that would last through a potential real attack.  Losing money is never good, but if you gotta lose money, make the most out of it that you can :)  I think if nothing else, some critical business points which are vulnerable to attack were exposed here.






I was discussing this very thing with another client of mine. I told him the story and he siad  "man that would seriously suck. You think they would tell the IT team about this to give them a heads up." Once he said that, i wondered for a moment. This pen test story was a great example of being prepared. No hacker is going to tell you " hey today i mihgt try and bust you up a we bit" So having that incident management team in place with some failsafes is a very good idea. Maybe the CFO didnt want his IT team to know about to make it more "real" But then again he claims he didnt know about it either. BUT to lose that kind of money on a pratice drill has gotta be painful.
Next

Return to Network Pen Testing

Who is online

Users browsing this forum: No registered users and 2 guests

cron
.
Powered by phpBB® Forum Software © phpBB Group.
Designed by ST Software