1. There are really two kinds of files, plain text and binary. As you have already realized, text files are easily recognizable. Binary files typically have a header and a footer element that identifies the file. The header or footer are just a series of characters that are going to be common to all files of that type. When you are looking for a file, you can search for a particular header to identify the beginning of a file. Identifying the end can be somewhat more challenging though. If the file doesn't have a footer, you can just cut chunks of a certain size and hope that the native software viewer can reconstruct the file using error correction.
For example, a windows executable has a header signature of 4D5A.
Like mambru said, data carving software will look for file headers. Typically you set the headers you are seeking and a target file size. The software will search for the header and copy out data chunks of the size you specified.
The above is assuming that you don't have access to the file system table. Most forensic software can interpret the file system type and its table of files and folders. Rather than relying on file signatures, the software will simply reconstruct the directory and file structure.
2. Some encryption protocols will include a header to identify itself. Some will not. In forensics, we get used to looking at data in hex. If we see a long stream of hex without any recognizable text, it could indicate encryption or some sort of encoding, but it isn't always the case.