There's no logical reason for suspecting Milton might be TG's brother. It's just that I felt that sensation when he grabbed him up. Okay. So, NOT brothers.
But I believe there's a deep pre-ZA connection. As I stated earlier, I think Milton has been agoraphobic. He fears the outside world. Telecommuting to work is a hint at this. Also, his fear of escaping with Andrea to the prison ("I don't 'know anyone' there.") I still believe Phillip may have been his front man in the past (at work) and is the one who got him to leave his apartment. And I think, perhaps, Milton was the brains and organization behind the grouping until Phillip gradually took over, using his charm to sway the people as the numbers and Milton's discomfort with those numbers grew. Milton was grateful that Phillip brought him out of hiding. He eventually took a back seat in planning, and now has little authority in decision-making. Just my theory. I kinda like it.
I get the feeling that there is some deeper relationship between Philip and Milton as well. Not that they are necessarily brothers but there seems to me to be something between these two guys. Not a romantic/physical relationship though; that is not the vibe I get.
Milton is not willing to give up on the Governor despite the torture chamber and whatever other atrocities he knows about, and despite having appeared to be wary of TG since we met them both. TG seems strangely uninterested in punishing Milton despite the fact that he knows Milton spilled the beans to Andrea, knew she was escaping and apparently burned the walkers.
It could be as simple as these two guys being the only two survivors of their original tiny group. Maybe they did know each other before the ZA, when Penny was still alive, before TG went over the edge. Maybe it's nothing. TIme will tell.