With many recent questions on “how do I exploit this?” or “if I do this on the network?” on the boards, this incident is worth thinking over before playing with a system or network that you don’t have written permission for.

I got a call from a friend, whom we’ll call him Dave, who was having some account lock out problems on his Windows domain. He was a bit spooked as the accounts were service accounts for backup, monitoring and SQL and they all locked out within minutes of each other. This stopped two major production systems in the middle of the working day and was causing him a stream of abusive phone calls on why “his” IT systems weren’t working and what the heck was going on.

Given Dave’s network was pretty well patched, they used a good account naming standards (so not accounts named backup, admin, etc) and every device had a decent AV on it, his admin senses were warning him something hoaky was going on. I directed him to look on his domain controller for the account lock out event id’s 539. Event ID 539 has the machine name, which means you can trace back the machine causing the lock outs. The machine name came back as XP, which was odd as the company machines are named after their asset numbers for accounting purposes. Pinging the machine’s name returned its IP address but no echo replies, so it appeared to be down. As I was asking questions, three more service accounts got locked out, all from workstation XP.

On a hunch, I asked him to check his DCHP server, look for the IP address there and tell me what the MAC address was. He found it quickly and noted that it looked different from the Dell machines MAC address details.
A quick look up on http://www.coffer.com/mac_find/ confirmed with was a VMware machine. Dave was somewhat disturbed by this as they are a windows only shop, including any virtualisation software. Dave’s LAN covers several floors and a large warehouse and has 16 switches, so we went with the old trick of searching the switches’ CAM tables for the MAC address to located which switch and port this IP address was connected to. As it happens the wiring diagrams were well maintained; the switch and offending port the MAC address had been connected was down one flight of stairs. Another three accounts locked out by the same machine name while we were doing this. A somewhat angry and confused Dave popped down stairs to see what was going on.

A now very excited Dave call back telling that a guy he’d never seen before was sitting at a spare desk, plugged in to the “evil” port, working on a shiny Mac laptop. He was on the way down to the warehouse to gather up some of the bigger lads to act as back up when he confronted the intruder.

Dave’s a big fan of the show 24, and imagination had got carried away with him. I told him to turn around, call his boss and check with reception if they knew this person was first. Reception proved to be more helpful than his boss, who had no idea. The “nice young man”, according to the receptionist, was a guest of the chief finance officer. Dave ran up the stairs to the top floor, burst in to the CFO’s office demanding to know who this guy was and why he was plugged in to Dave’s network.

The shocked then annoyed CFO stared at the wild-eyed, panting Dave and calmly replied he was a consultant doing some testing highlighted in the last audit. Dave, bravely attempted to not get himself fired on the spot, blurted out that consultant was the one who’d just stopped the inventory database critical to the warehouse operations. The CFO brain connected that this was something bad and, with Dave following, rushed down to the consultant’s desk and order him to cease all activity.

Dave had all the accounts unlocked and all the services running again within the next hour. He was asked to sit in a meeting with the CFO, his boss, the consultant and the consultant’s boss. The angry CFO was demanding answers on what had happened. The two from the consultancy exchanged looks and hand over the penetration testing contract and scope of work. All signed and agreed by the CFO.

To cut a long and embarrassing (for the CFO) story short, new auditors had recommended a penetration test be carried out to assess the internal systems as IT systems where now critical to the business. The CFO decided not to involve or inform anyone from IT, against the penetration testing company’s objections, as he always dealt with these types of financial matters. He ordered an extensive testing program including password attacks, all to be carried out on site and during office hours. Dave looked over the scope of work and was horrified what the CFO had chosen, the document clearly stated some of these tests carried risk, but had been signed off on.

The outcome from this was
1)   The business  had a three hour outage in the middle of its working day when the consultant was in attacking mode by attempting password brute forcing on service accounts
2)   The business estimated losing half a million dollars during those three hours
3)   The two day pentest engagement was cancelled and a more sensible scope of work was drawn up
4)   IT was included in on any and all audit requirements
5)   The CFO got his wrist slapped.

Draw what you will from this tale, but always, always have written permission from someone in the business authorized to give it before even think about pen-testing a system.

I was waiting to hear that they didn't have the right paperwork in place.

Good real life example of getting ducks in a row.

Great write-up, thanks for sharing it with us. Personally I think it demonstrates some very important points which should be considered when performing any kind of pentest to a company. I am sure the CFO learned his lesson now, in quite an unpleasant but rememberable way though.

apollo has a great point here.  In my full-time gig (my non-security / non-pentesting gig,) the customers I support consider an outage as loss of $5 million every 15 minutes they are down, during any outage.  So time adds up quickly, and it's very important that they look at the overall results from the first test, analyze their response methodology and time to recovery, and make the most of their investment by modifying their incident response / recovery plans.  Too often, I see stupid outages, which take excessive amounts of time to recover, unnecessarily, solely because of poor planning, and in no way as a result of lack of training or knowledge.  And I've seen similar to this, when my customers (from said gig) have pentesting and analysis teams do work for them, too.  If it weren't for the fact that, in that job, I'm prohibited from engaging on that side of their activities, I'd just love to politely kick someone in the backside, for lack of a better term, and wake them up.  (BTW, doesn't mean that I don't drop subtle hints to said customers, under the radar...   )

It's very important, as pentesters, that we analyze these types of situations and outcomes, and use them to promote positive, forward-thinking and proactive activities for our customers, so that in the end, they remain as safe as possible, remain stable, maintain uptime, and truly reap the benefits of our service.  A good pentest should not be limited to solely showing faults and holes, but rather, should encompass good business practices and promote cooperation, both with, and within, our customers, as well.

Thanks, What90, for a good reminder, both of how NOT TO do something, but also opening a good discussion of how TO do things better.

perhaps the takeaway from that experience should have been to not be able to lock out a service account that causes that amount of loss per hour. instead create a rule in their SEIM to alert on failed logins for those accounts as that should NEVER happen.

DANG. We had a very similar experience on our network BUT the only diff was that it was not a pentest. some punk kid was trying to brute force our network because he was able to get the username file from our server using the null session with dumpsec tatic. Luckily we make sure our users have to follow our password policy. Still was annoying to get into work at 6 in the morning and have 33 accounts locked out.

Anyway. awesome post. Ill be sure to remember this for future use and recommendation.

Yeah, absolutely. Not only gets them to actually read stuff but also getting them thinking about the dangers that are really out there. The "bad guys" don't have any rules to play by.

I was discussing this very thing with another client of mine. I told him the story and he siad  "man that would seriously suck. You think they would tell the IT team about this to give them a heads up." Once he said that, i wondered for a moment. This pen test story was a great example of being prepared. No hacker is going to tell you " hey today i mihgt try and bust you up a we bit" So having that incident management team in place with some failsafes is a very good idea. Maybe the CFO didnt want his IT team to know about to make it more "real" But then again he claims he didnt know about it either. BUT to lose that kind of money on a pratice drill has gotta be painful.