On their way south, Sam and Gilly (and baby Sam!) stopped for the most awkward family dinner known to man. There was a seriously uncomfortable conversation with Sam's horrible father, and it led to Sam and Gilly leaving the Tarly family home in the middle of the night. Where is Sam going on Game of Thrones?
Sam and Gilly first exited Castle Black back in the Season 5 finale, with Sam intending to follow his original dream to become a maester and serve the Night's Watch in that way. A maester is a highly educated scholar; every great House in Westeros has one. After the Night's Watch lost theirs (the aged Aemon Targaryen), it was the perfect opportunity for Sam to go off and study before taking over the post. That's still Sam's goal: to head to what is essentially Maester University and learn all he can learn.
However, on the way there, Sam intended to drop off Gilly and Lil Sam with his family so they would be taken care of while he was busy. The problem was that Sam's awful dad proved to be even more awful than anyone could have guessed, and Sam had finally had enough.
Sam's mother and sister proved to be as sweet as Sam himself and they were very welcoming to Gilly, but his father was another story. For a refresher on just how terrible Lord Tarly is: the reason he sent his firstborn son, Sam, to Castle Black instead of allowing him to inherit like he was supposed to is because he was deeply disappointed that Sam was bookish and peaceful instead of belligerent and sporty. He even threatened to kill Sam if he didn't agree to leave – so it's no surprise that he's far from pleasant when he gets to see Sam again.
Lord Tarly spends almost all of family dinner picking on Sam for his decision to become a maester, his scholarly interests, and his weight, but he takes it too far when he insults Gilly for being a wildling. Sam isn't able to stand up to his father in the moment because he's so fearful that doing so will leave Gilly without a place to stay, but Sam gets the last word: when he decides to take Gilly and Lil Sam with him on the way to the maesters' Citadel, he takes his father's Valyrian steel sword with him.
So it looks like Sam will be continuing his journey with his family in tow, which could make things difficult when he arrives at his destination. Sam will be traveling to the maesters' Citadel in Oldtown, but this might leave Gilly alone and without protection while Sam undergoes his training. Gilly can probably take care of herself just fine, but it's still putting her in a rather precarious position.
But hey – at least they're together.