You are conflating what SSH does with what the terminal you're running it in does. SSH barely knows what terminal emulation is. You can literally pipe binary data into it at one end and receive that data into a program at the other end. That's what programs like scp and rsync do. They launch ssh and use it as a data channel.