During an episode of the That Scene with Dan Patrick podcast, the actor opened up about the relationship he has built with Brown, 17, since the series first began back in 2016.
"Millie and I have always had sort of a special relationship because I knew her when she was so young. I knew her before any of this big fame hit," he said.
Harbour, 46, plays Jim Hopper, the Hawkins, Indiana police chief as well as the adoptive father to Brown's Eleven.
"I have a real protective feeling for her. I have a real, like, worry," he continued. "I worry about her and the fame and all that she has to struggle with. And I've just always felt this kind of deep fatherly affection for her."
Harbour also opened up about his character's arc in the upcoming season, saying he brings Hopper "back into the warrior realm" rather than focusing only on his parenting duties.
"One of the interesting things about Hopper this season — although you do see a lot of that fatherly stuff is — I did want to take him back into the warrior realm because there's something as you get to be the dad role where, you know, it's a bit like the dad gene," he said.
Netflix David Harbour in Stranger Things
"You start to become less of a viable presence in the world," Harbour continued. "They're more like, 'Oh, dad' — like someone who people are humiliated by, and like, I'm not quite ready for that. I still want to be, like, a presence."
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
Netflix announced in October that Stranger Things had resumed production on season 4 after the COVID-19 pandemic halted filming earlier in the year. The show was a month into production on the upcoming season when the pandemic hit the U.S. in March 2020, with the scripts polished and ready to go, PEOPLE previously reported.
Set to return for season 4 alongside Harbour and Brown are Winona Ryder (Joyce Byers), Gaten Matarazzo (Dustin Henderson), Finn Wolfhard (Mike Wheeler), Noah Schnapp (Will Byers), Caleb McLaughlin (Lucas Sinclair), Natalia Dyer (Nancy Wheeler), Charlie Heaton (Jonathan Byers), Maya Hawke (Robin Buckley), Sadie Sink (Max Mayfield), Joe Keery (Steve Harrington), Priah Ferguson (Erica Sinclair) and Cara Buono (Karen Wheeler).
In an interview with PEOPLE in December, Harbour teased that in the new season, audiences will "see a lot of your favorite characters doing the thing that you love them for ... but you'll also see them twist it and turn it into other colors and other flavors."
Harbour also said that fans will "get to see an entirely other color" of Hopper in season 4, which does not yet have a premiere date. "This season is my favorite because he really is going to show you some new colors and what he's really made up of," he said.