I agree and apparently so do the authors of books on the topic. Both Ed Skoudis' book Counter Hack Reloaded
and Michael Gregg's book CEH: Exam Prep
both have in depth coverage of TCP/IP. In fact, you can read that specific chapter of Gregg's book here on this site by clicking the title link.
Your suggestion of being able to visualize the packet flow is a great one. Did you get a chance to look at the animation video I posted on the travles of a packet named Warriors of the Net
Whether it is looking at Snort or firewall logs to figure out what has happened, doing real time analysis using Wireshark or scanning using Nmap, a good understanding of TCP/IP is crucial.
PS - Hey Kev, you're making a nice push for the Free Monthly Giveaway. Keep up the good work.