On the Season 6 episode "Blood of My Blood," Samwell, Gilly, and Little Sam finally arrive at his home in Horn Hill. They don't get the warmest welcome from Lord Tarly, and his vitriol isn't solely directed at Sam. So why does Sam's father hate Gilly on Game of Thrones? He may hate her even more than he hates his own son.
Basically, Lord Tarly is hella racist against wildlings (in addition to the general fatphobia and toxic masculinity directed towards his firstborn). Gilly learns this as they approach Horn Hill: that Sam never told his family she's a wildling. In fact, Lord Tarly shipped Sam off to The Night's Watch specifically so he could "make a man out of [himself] killing some bloody wildlings."
"My father doesn't like wildlings," he admits, forcing her to promise that she won't tell Lord Tarly her origins.
Over dinner, though, Gilly slips, and, in trying to defend Sam's bravery, accidentally mentions that he killed a White Walker on their way down to Castle Black.
"Where did he kill a White Walker?" Lord Tarly asks "Where are you from? How'd you come to meet my son?"
Gilly defiantly announces that The Night's Watch found her north of The Wall.
"North of The Wall?" he sputters. "You're a wildling. The Seven Kingdoms have waged war against these savages for centuries and here I sit hosting one in my hall, thanks to my son."
Lord Tarly continues:
"I took you for a Mole's Town whore when I saw you and I made my peace with that. Who else would have him? I overestimated him. No, it was a wildling whore who seduced my son. Is this you getting back at me, boy? Bringing that to my table and making me dine with it? And you got what you were after, didn't you? A bastard? A half-breed bastard."
He's about to rescind her invitation to his home, but, surprisingly, Lady Tarly interrupts him and whisks Gilly away. Lord Tarly agrees to let Gilly stay and work in the kitchens while Little Sam may be raised at Horn Hill. Samwell, however, is no longer welcome.
Of course, Sam gets a grip, grabs the family heirloom Valyrian steel sword, and escapes Horn Hill in the middle of the night with Gilly and Little Sam in tow. It's unclear where they're headed just yet, but it seems like they're at least determined to stay together as a family.