A cautionary tale for Penetration testers on live networks

Viewing 15 reply threads
  • Author
    Posts
    • #4754
      What90
      Participant

      With many recent questions on “how do I exploit this?” or “if I do this on the network?” on the boards, this incident is worth thinking over before playing with a system or network that you don’t have written permission for.

      I got a call from a friend, whom we’ll call him Dave, who was having some account lock out problems on his Windows domain. He was a bit spooked as the accounts were service accounts for backup, monitoring and SQL and they all locked out within minutes of each other. This stopped two major production systems in the middle of the working day and was causing him a stream of abusive phone calls on why “his” IT systems weren’t working and what the heck was going on.

      Given Dave’s network was pretty well patched, they used a good account naming standards (so not accounts named backup, admin, etc) and every device had a decent AV on it, his admin senses were warning him something hoaky was going on. I directed him to look on his domain controller for the account lock out event id’s 539. Event ID 539 has the machine name, which means you can trace back the machine causing the lock outs. The machine name came back as XP, which was odd as the company machines are named after their asset numbers for accounting purposes. Pinging the machine’s name returned its IP address but no echo replies, so it appeared to be down. As I was asking questions, three more service accounts got locked out, all from workstation XP.

      On a hunch, I asked him to check his DCHP server, look for the IP address there and tell me what the MAC address was. He found it quickly and noted that it looked different from the Dell machines MAC address details.
      A quick look up on http://www.coffer.com/mac_find/ confirmed with was a VMware machine. Dave was somewhat disturbed by this as they are a windows only shop, including any virtualisation software. Dave’s LAN covers several floors and a large warehouse and has 16 switches, so we went with the old trick of searching the switches’ CAM tables for the MAC address to located which switch and port this IP address was connected to. As it happens the wiring diagrams were well maintained; the switch and offending port the MAC address had been connected was down one flight of stairs. Another three accounts locked out by the same machine name while we were doing this. A somewhat angry and confused Dave popped down stairs to see what was going on.

      A now very excited Dave call back telling that a guy he’d never seen before was sitting at a spare desk, plugged in to the “evil” port, working on a shiny Mac laptop. He was on the way down to the warehouse to gather up some of the bigger lads to act as back up when he confronted the intruder.

      Dave’s a big fan of the show 24, and imagination had got carried away with him. I told him to turn around, call his boss and check with reception if they knew this person was first. Reception proved to be more helpful than his boss, who had no idea. The “nice young man”, according to the receptionist, was a guest of the chief finance officer. Dave ran up the stairs to the top floor, burst in to the CFO’s office demanding to know who this guy was and why he was plugged in to Dave’s network.

      The shocked then annoyed CFO stared at the wild-eyed, panting Dave and calmly replied he was a consultant doing some testing highlighted in the last audit. Dave, bravely attempted to not get himself fired on the spot, blurted out that consultant was the one who’d just stopped the inventory database critical to the warehouse operations. The CFO brain connected that this was something bad and, with Dave following, rushed down to the consultant’s desk and order him to cease all activity.

      Dave had all the accounts unlocked and all the services running again within the next hour. He was asked to sit in a meeting with the CFO, his boss, the consultant and the consultant’s boss. The angry CFO was demanding answers on what had happened. The two from the consultancy exchanged looks and hand over the penetration testing contract and scope of work. All signed and agreed by the CFO.

      To cut a long and embarrassing (for the CFO) story short, new auditors had recommended a penetration test be carried out to assess the internal systems as IT systems where now critical to the business. The CFO decided not to involve or inform anyone from IT, against the penetration testing company’s objections, as he always dealt with these types of financial matters. He ordered an extensive testing program including password attacks, all to be carried out on site and during office hours. Dave looked over the scope of work and was horrified what the CFO had chosen, the document clearly stated some of these tests carried risk, but had been signed off on.

      The outcome from this was
      1) The business  had a three hour outage in the middle of its working day when the consultant was in attacking mode by attempting password brute forcing on service accounts
      2) The business estimated losing half a million dollars during those three hours
      3) The two day pentest engagement was cancelled and a more sensible scope of work was drawn up
      4) IT was included in on any and all audit requirements
      5) The CFO got his wrist slapped.

      Draw what you will from this tale, but always, always have written permission from someone in the business authorized to give it before even think about pen-testing a system.

    • #29740
      Ketchup
      Participant

      Chris, that’s a great story.  Thanks for posting it, since there are many lessons to be learned there.  Coming from the consulting side of things, I was relieved to read that the pentesters had proper contracts in place. 

    • #29741
      rattis
      Participant

      I was waiting to hear that they didn’t have the right paperwork in place.

      Good real life example of getting ducks in a row.

    • #29742
      What90
      Participant

      According to Dave, the penetration testing guys were true professionals and had warned the CFO repeatedly, both verbally and written, on the risks he was taking. The CFO used the auditor recommendations as a have to do list.

      As another display of  business awareness, the reason that only three accounts where being locked out at every five minutes was the consultant being as careful as he could. Rather than just running a brute force attack at the entire enumerated AD user names.

      One really nice thing I liked, if Dave had appeared with his heavies, the consultant had a signed copy of the contract and all the contact numbers for the CFO with him at all times during the engagement.

      Gold star to the Pentest team and unfortunately stock options for the CFO….

    • #29743
      UNIX
      Participant

      Great write-up, thanks for sharing it with us. Personally I think it demonstrates some very important points which should be considered when performing any kind of pentest to a company. I am sure the CFO learned his lesson now, in quite an unpleasant but rememberable way though.

    • #29744
      apollo
      Participant

      Out of curiosity, did the pen testers recommend any strategic changes to your incident response procedures or any additional procedures to put in place in case this happens again ?  This is an excellent example of how having an incident response team with the proper professionals on it could have probably gotten things resolved faster.  You post this as a cautionary tale, with good reason, but it seems like there could have been some great positives come out of this that would last through a potential real attack.  Losing money is never good, but if you gotta lose money, make the most out of it that you can 🙂  I think if nothing else, some critical business points which are vulnerable to attack were exposed here.

    • #29745
      hayabusa
      Participant

      apollo has a great point here.  In my full-time gig (my non-security / non-pentesting gig,) the customers I support consider an outage as loss of $5 million every 15 minutes they are down, during any outage.  So time adds up quickly, and it’s very important that they look at the overall results from the first test, analyze their response methodology and time to recovery, and make the most of their investment by modifying their incident response / recovery plans.  Too often, I see stupid outages, which take excessive amounts of time to recover, unnecessarily, solely because of poor planning, and in no way as a result of lack of training or knowledge.  And I’ve seen similar to this, when my customers (from said gig) have pentesting and analysis teams do work for them, too.  If it weren’t for the fact that, in that job, I’m prohibited from engaging on that side of their activities, I’d just love to politely kick someone in the backside, for lack of a better term, and wake them up.  (BTW, doesn’t mean that I don’t drop subtle hints to said customers, under the radar…  😉 )

      It’s very important, as pentesters, that we analyze these types of situations and outcomes, and use them to promote positive, forward-thinking and proactive activities for our customers, so that in the end, they remain as safe as possible, remain stable, maintain uptime, and truly reap the benefits of our service.  A good pentest should not be limited to solely showing faults and holes, but rather, should encompass good business practices and promote cooperation, both with, and within, our customers, as well.

      Thanks, What90, for a good reminder, both of how NOT TO do something, but also opening a good discussion of how TO do things better.

    • #29746
      BillV
      Participant

      Wow, good story. Definitely a great heads up for everyone.

      It reminds me of something an instructor had said in one of my courses about including some “out there” items in the documentation of what tests to conduct (e.g., following employees home, kidnapping, etc.) just to see if the person signing off on it actually reads it or just checks all tests and signs off. Then of course call said person up, ask whether they read through the paperwork and whether they really wanted to allow employees to be kidnapped. Crazy, sure, but certainly realistic if the information is important to someone.

    • #29747
      Anonymous
      Participant

      perhaps the takeaway from that experience should have been to not be able to lock out a service account that causes that amount of loss per hour. instead create a rule in their SEIM to alert on failed logins for those accounts as that should NEVER happen.

    • #29748
      former33t
      Participant

      Wow.  I don’t even know where to start here…

      Bill, I love the idea you shared about the outlandish ideas.  I’ll be sure to use this both in consulting and at my normal job.  My boss routinely ok’s things that I’m positive he doesn’t comprehend (and as a result doesn’t read).  A reality check might force a positive change.

      ChrisG, you are right on the money about not locking out service accounts.  How you choose to deal with the problem of service account login failures is an administrative policy choice, but I have never seen locking them out have positive outcomes.  Instead of the one application using outdated credentials, everything that depends on the service account is down…

      What90, I hadn’t even appreciated the fact that the pentesters were taking the time to make sure they didn’t lock out the whole domain in one fell swoop.  Good for them.  I’ll make sure that I incorporate that into my next security assessment if I’m asked to attempt brute forcing of user accounts.

    • #29749
      LT72884
      Participant

      DANG. We had a very similar experience on our network BUT the only diff was that it was not a pentest. some punk kid was trying to brute force our network because he was able to get the username file from our server using the null session with dumpsec tatic. Luckily we make sure our users have to follow our password policy. Still was annoying to get into work at 6 in the morning and have 33 accounts locked out.

      Anyway. awesome post. Ill be sure to remember this for future use and recommendation.

    • #29750
      What90
      Participant

      @Apollo

      They got some great recommendations, but none of them transitioned off the report pages. Dave’s company is a small place, around 250 staff, the IT “team” consists of Dave’s boss (a non tech manager handed running IT as he showed some interest), Dave, and a junior desktop guy. Dave’s incident handling process was to ring a friend (me, in this case) and ask for help.

      I’ve seen the top ten objectives, management set, IT are to achieve this year and security wasn’t one of them. In fact, security is only ever raised by the auditors who seem obsessed with IDS and what type AV they have, rather than if it’s actually being used correctly 🙂 It’s the normal excuse of we don’t have the money to spend on security. The pentest was seen as a one off problem. Somewhat myopic…

      Dave and the team have got a couple of positive outcomes from the event, but the overall postures is still reactionary for security.

      @ChrisG
      Account lock outs for all accounts after 5 failed attempts is on the auditors check list. It’s a best practice apparently….

      I pointed Dave to OSSIM http://www.alienvault.com/community.php?section=Home as a easy way to watch the network, but he isn’t given the time to set it up. Instead he’s currently been directed to look at Ms’ SCOM product for better monitoring of his network.

    • #29751
      BillV
      Participant

      @former33t wrote:

      Bill, I love the idea you shared about the outlandish ideas.  I’ll be sure to use this both in consulting and at my normal job.  My boss routinely ok’s things that I’m positive he doesn’t comprehend (and as a result doesn’t read).  A reality check might force a positive change.

      Yeah, absolutely. Not only gets them to actually read stuff but also getting them thinking about the dangers that are really out there. The “bad guys” don’t have any rules to play by.

    • #29752
      hayabusa
      Participant

      @BillV wrote:

      Yeah, absolutely. Not only gets them to actually read stuff but also getting them thinking about the dangers that are really out there. The “bad guys” don’t have any rules to play by.

      Amen!  Absolutely the case.  The bad guys could care less about any rules of engagement, recovery plans, etc.  They’re going to attack, regardless.  All of these things need to be taken into account, and should be clearly defined, and noted to the customer, so that they understand both risk and reward for testing and remediation.  This is why you have to put in the extra time, to show why pentesting services are valuable to customers, and to get mgmt to understand their own company’s situation(s).

    • #29753
      LT72884
      Participant

      @apollo wrote:

      Out of curiosity, did the pen testers recommend any strategic changes to your incident response procedures or any additional procedures to put in place in case this happens again ?  This is an excellent example of how having an incident response team with the proper professionals on it could have probably gotten things resolved faster.  You post this as a cautionary tale, with good reason, but it seems like there could have been some great positives come out of this that would last through a potential real attack.  Losing money is never good, but if you gotta lose money, make the most out of it that you can 🙂  I think if nothing else, some critical business points which are vulnerable to attack were exposed here.

      I was discussing this very thing with another client of mine. I told him the story and he siad  “man that would seriously suck. You think they would tell the IT team about this to give them a heads up.” Once he said that, i wondered for a moment. This pen test story was a great example of being prepared. No hacker is going to tell you ” hey today i mihgt try and bust you up a we bit” So having that incident management team in place with some failsafes is a very good idea. Maybe the CFO didnt want his IT team to know about to make it more “real” But then again he claims he didnt know about it either. BUT to lose that kind of money on a pratice drill has gotta be painful.

    • #29754
      j0rDy
      Participant

      correct me if im wrong, but i see some flaws on either the CFO’s side as the pentesters side. Dave acted correctly from the moment he discovered the “hack”. The CFO should have never given permission to attack the availablity of the infrastructure, and the pentesters should have never kicked up that much dust to get detected. however, when compromising the availablity of the infrastructure, youre pretty sure you will be detected.

Viewing 15 reply threads
  • You must be logged in to reply to this topic.

Copyright ©2021 Caendra, Inc.

Contact Us

Thoughts, suggestions, issues? Send us an email, and we'll get back to you.

Sending

Sign in with Caendra

Forgot password?Sign up

Forgot your details?