Where Are The Rest Of The Towns?

So When Glenn had his first kill. he looked up and saw about 20 photos on the wall with guys heads all bashed in.. Negan said they only pick one to make an example of. so if they live in a factory where nothing grows.. they had to been extorting many towns. to make it work.. Also he had a bunch of wife's that he said he would take after he killed their man.. 

 

So why don't Rick and the funky bunch just move people to the 3 towns we know of and stop supporting the factory.. ?

Less sets to build for shooting.

Lets not forget the lady with the key to the future. and the Helicopter.

Saviors actually wiped out everyone from the Library, which I assume was another community. I also think some of the communities were too small and actually absorbed into the population of the Sanctuary. There may be more, but the only one that's been found so far was Oceanside.

What about the time when Michonne kidnapped that lady Savior and made her drive to there camp? it was out in some field. and they just turned around and left.

As I recall Michonne saw the sheer size of The Sanctuary then turned around to tell everyone about it.