The reason why they fail to sanitize user-input globally, is because of more than one developer, stress on the developers, and thereby an insecure development lifecycle.
Initially, the script may have been secure, but over time when it has been extended and extended, reaching ten-thousand lines of code (in several different files) or more, without proper quality control, it may eventually become insecure.
Often, a "catch and sanitize all" function, is not the best thing to do, as it may break the visual formating of input and output, or not function correctly. But just like simple stack overflows or "Remote File Inclusions" are hardly seen anymore in well known apps, while "off-by-one" and XSS is often hard to completely secure against in huge applications, that weren't secure from the start and hasn't had a completely safe development lifecycle.
There are of course, occasionally new attacks, that suddenly spawns and affects pretty much all applications, such as Click Jacking, Cross-Site Request Forgery, and more. Vulnerabilities, or "features" that most, didn't consider a vulnerability until they were perhaps, widely abused or realised to be a risk.
This depends on the case. But as the examples above, were non-persistent GET-request based XSS items, most likely.
By knowing different WAF's, webservers, and common applications, you can tell if it's most likely "this" or "that". But there's no way to be 100% sure, as the way a WAF filters input / output, can easily be integrated into a web application as a custom function, or for that sake, a webserver.
It's often not important to find out where the filter / sanitization is done, only if you know there's a publicly available copy of the source code somewhere, so you can take a look at the function preventing the XSS.
Yes. Even though it requires user-interaction / social engineering, but I've been successful with that several times (for legal harmless purposes of course). So it is possible, it just requires the right attack vector, where the human element plays a role. Of course, with persistent XSS, there's also a human role, but if there's no protection against the XSS attack in the browser, then it's a very limited role except that the attacker waits for the user to trigger the exploit / malicious piece of code.
What can do be done? Here's an example: http://www.exploit-db.com/vbseo-from-xs ... php-shell/
That's custom work of course, but it works, and it was fun to create. (It took some research and time of course.)
XSS is always executed client-side. It may force a user, to execute PHP code somehow via a feature on the target website, or compromise the user's browser and execute system commands, but it (the XSS payload) is still, executed client-side.
I think it would be a good idea, if you researched a bit about how a webserver / HTTP server functions, not as in how it serves data in detail. More like: Which ports, which processes are used (php, mysql, apache, as an example), how do these work together, what happens when you request e.g. index.php on a webserver (you should read the raw http request to see it with your own eyes), and so forth. This is "simple" (for me), but may sound advanced even though it is not, once you've tried it out yourself.
The best way to learn, is:
1. Set up a webserver, such as XAMPP on Windows where everything is done for you, or a LAMP yourself. LAMP means Linux Apache Mysql PHP. The last, can also be Python or Perl of course.
2. Write a few small scripts like <?php echo "Hello!"; ?> and test if they work. (Make sure to place them in the right directories.
3. Access these scripts, in your webbrowser. Do they work? Yes? If so, install an intercepting proxy like Burp Suite Free, and redirect traffic from e.g., FireFox into that, and then hit F5 on the page where your own script is that is served from your own webserver, and then, view how it looks like.
It's raw text! And if you take a look at most of the headers, it will make sense, at least, somewhat to begin with. Some things may not make sense. Over time, you'll suddenly find out, that it's a lot easier than it appears to be.
7) Yes, with XSSQLI. http://www.xssed.com/article/31/The_Beg ... de_to_XSS/
(MUST READ!) You also need to understand in depth, how all this plays together and if you have to use XSS to perform SQL Injection or not. For example, if a SQL Injection is only located in an administration panel, and there's an XSS or CSRF issue in that panel, you should use these (XSS and / or CSRF) vulnerabilities, to trick a user into performing the SQL Injection for you, which you're not allowed to as you don't have access to the administration panel. That's just an example of course.
Send a targeted user a maliciously crafted link, via e-mail. (If HTML is used, the link can be masked somewhat, including the use of a few "tricks".)
A (soon) happy new years!