you can use the Windows 2008 functional level, you only need to go mixed if you are running Domain controllers that are not 2008. Going to the 2008 domain functional allows him to configure more of the advanced features available in 2008 domains.
As far as getting them to talk, you may just need to configure some routes on the network. I had to do this at one point and all I did was create a route map on my linksys broadband router to tell clients where to go for the lab IP range. Ideally you should restrict this communication to a single device though. The security testing lab should be configured away from your regular systems and maybe only allow it out to the internet, which can be accomplished using a gateway of sorts on the lab, either a physical box with 2 NICs passing the traffic or a separate router connected to your main internet gateway.
So if you are using your BT system outside of the lab, you can create a route to point to the NIC IP you configured on your 2008 box that has access to the physical LAN. On the 2008 box you will need to have RRAS configured so it can pass the traffic back and forth. On the BT system you can create a static route for the VM lab LAN, the gateway would be the physical LAN NIC of the 2008 server.
As for the questions at hand...
On the 2008 box, if you made it a DC already, best to scrap it and revert back to the base before promoting it. Get the hardware piece settle first. Also I would advise configuring a separate 2008 server for RRAS. RRAS will cause some headaches on a domain controller.
- STOP! Create another 2008 box, if you want to avoid the patching and installation, clone your current one if it has a clean snapshot. Then proceed to building your DC. It will act as a regular system in your LAB. I advise building a separate 2008 box for RRAS services.
- On the RRAS box, Before firing it up, install a second virtual NIC. Configure the NIC as a bridge, that way it will connect directly to the main LAN through your host adapter. Don't worry about the static IP yet.
- Set the original VM adapter as Custom and pick one of the options. NAT will just cause it to also connect to the physical network. Host Only will most likely give you routing issues.
- Power up the 2008 box and begin the static configurations for the IP addresses. Assign the Bridged adapter an IP on your physical LAN that is not in use, I typically choose IPs not currently in the DHCP range. For DNS just configure whatever your default one is for the physical LAN.
- Configure your custom NIC using an IP that is not part of your physical LAN subnet. If you use 192.168.0.0/24 on the physical LAN, use 10.0.0.0/24 or 172.16.0.0/24 on the lab LAN.
- Now back to the DC build, set the network adapter to use the custom VMNet# you chose for the RRAS box. Fire it up.
- Ensure the system is fully patched and configure the network settings accordingly (using the lab subnet). For DNS set the primary to itself for now.
- If you cloned this from your first 2008 ensure the hostnames are different and proceed to promoting it to a new DC in a new forest.
- Once completed, you should be all set for the DC setup. You may need to tweak DNS to have some forward lookup zones. They will probably not work until you finish your RRAS setup.
- Go to the RRAS box and try to join it to the new domain (make sure you can see the new DC through some Ping tests) oh and configure the DNS server on the lab NIC to use your new DC's IP. Once it is joined you can continue with finishing the setup.
- Install the RRAS services and the main service you will want to use is the routing portion. I have setup RRAS in a while so I forget the steps. I know doing it on a member server makes things happier, also it is how you would see it in a real environment, though in many cases it is even a standalone system.
- Back to DC, install DHCP and configure. Also make sure the gateway is the RRAS server and routing is working properly.
- If all seems good, then you should be able to start adding client systems to the network.
- As for your backtrack system, adding a route for the LAB subnet to point toward the physical LAN NIC for the RRAS box should work.
This should get you in the right direction, you may have some hiccups but it is probably the better way to do this. I've gone and installed RRAS on a DC after-the-fact and it gave me headaches with the NIC configuration. 2008 RRAS likes having 2 NICs, it will take the first it sees and screw up whatever other configs are present.