Well in general unless you are doing some type of driver or kernel programming, you aren't going to mess up your computer. If you code a program incorrectly, it will crash but won't affect the system as a whole (OK, there are some ways you could hose up your system, but you'd have to really be trying).
I'm assuming you're running Windows, so about the only built in development tool is going to be debug. What software you need depends on what programming language you want to learn - personally I would suggest learning C at first, but there are plenty of other languages out there. Interpreted languages are easier to learn, such as Perl and Python.
For writing programs in C on Windows I prefer to use Dev-C++ from bloodshed.net; very straight forward for writing simple programs, but has more powerful features when I want them. You can also get Windows interpreters for Perl and Python from activestate.com.