NSE's are written in LUA. The biggest challenge when I was working on NSE devel is that LUA is missing some things I really wanted. Read a basic LUA tutorial, and then on the Nmap site in the docs there's a section dedicated to working with NSEs. Once you understand the basics, you can look at the stock stuff to figure out more. The big thing is that libraries are changing pretty quickly and growing. Make sure when you are doing devel, you are using the latest nmap release, otherwise you are going to be missing a ton of libraries and other examples.
hope this helps
CISSP, CSSLP, MCSE+Security, MCTS, CCSP, GPEN, GWAPT, GCWN, NOP, OSCP, Security+