So yea! Congrats, you've taught English in ABC country for a year or two and then go home to visit your family. While you've been gone they've gotten jobs as engineers, designers, nurses, marketing directors, etc, and you're "just an English teacher".
Another possibility is that you decide to teach English for a year or two and then go back home in order to look for a job. If you've learnt a language while you're gone then that will help you out when you interview. However, most people don't. When you go to interviews and the interviewer asks you what you've done for the past couple of years you're going to have to explain that you taught English. Unfortunately, while there are many serious teachers out there many employers think of English teachers as backpackers who are trying to escape the real world.
Lack of Stability
One moment the place where you're working at is great, the next it's horrible. New admin, budget cuts, caps on how long you can stay at your job all create instability. Need an example? Take a look at HCT in the UAE. Once known as one of the best places to work at in the UAE, out of the blue they started getting rid of people right and left and only giving them a month to find another job.
Even if your place doesn't have these issues it's pretty rare to sign a contract that lasts more than 3 years. The majority of them are only a year longer, maybe two, three years if you're lucky. Still, three years is nothing compared to jobs back home that don't have an ending date.
Granted in some places such as Korea, the Middle East, and Africa, teachers can earn decent money and have good benefits. However, compared to other jobs back home it's not that much. Compared to international school teachers, it's a lot less and compared to what expats make in other careers, it's a pittance.
Many teachers may not get those good jobs in Korea, the Middle East, or Africa and instead are happy with $500 a month in China, because they're told that it's twice as much as the local salary. Comparing what a foreigner needs and what a local needs is illogical. While living in real China and earning $500 a month might seem ok for a bit, the novelty quickly wears off.
Lack of Advancement
Some teachers move up the TEFL ladder and get jobs as DOSes, teacher trainers, or consultants, many teachers remain simple teachers for the majority of their careers. Even if you do get the chance to advance your career, chances are you will always have to answer to a local employer who may have a lot less experience and qualifications than you do.
Lack of Serious Co-Workers
As you advance your career and move on to better jobs, you'll find that you'll work with more serious teachers. However, at the beginning of your career, you may find teachers who are more worried about what drink they'll have at the pub or where they'll take their date on the weekend.
Even if you do get a job at a good university or school, you'll probably still run into some odd teachers. Due to the ease to get a job and the laid back lifestyle, TEFL attracts some very weird people.
All of the above leads to the rose coloured glasses slowly coming off. Living in a foreign country, not knowing the language, the culture, and always being seen as a foreigner tends to wear off. Sure, some people adjust, learn the language, marry locals, even become citizens, yet the majority of them will still be seen as outsiders and never fit in completely.
Don't get me wrong, I enjoy teaching, but I've experienced all of the above and can see why some teachers get out of teacher or stay and simply become hardened, jaded teachers. TEFLing can be great if you land a good job, people have lived well, saved money, bought houses outright and retired young. If you do it correctly it can be very lucrative.