Cumberbatch plays Sherlock Holmes in the BBC drama, while Mr Freeman portrays his assistant, Dr Watson.
When asked about a possible fifth series last month, Freeman said people's expectations are "not fun anymore."
In an interview with the Daily Telegraph, Cumberbatch disagreed, labelling his colleague's unhappiness "pathetic".
When asked if fan fervour had spoiled his joy in Sherlock, he told the newspaper: "It's pretty pathetic if that's all it takes to let you not want to take a grip of your reality.
"What, because of expectations? I don't know. I don't necessarily agree with that."
Freeman said he didn’t enjoy working on the series anymore due to the hype surrounding it.
He told the newspaper: "Being in that show, it is a mini-Beatles thing. People's expectations, some of it's not fun any more. It's not a thing to be enjoyed."
Cumberbatch, who is promoting his role as Doctor Strange in Marvel's Avengers: Infinity War, said he was grateful for the Sherlock fans' support.