James Caan apparently went somewhat method filming Elf

Elf isn’t known as one of the greatest holiday movies of all time just for its claymation narwhal or heart-attack-waiting-to-happen spaghetti dinner—though those things obviously help. The real key to Elf’s success is its larger-than-life performances, specifically propped up by the ingenious idea to bring a serious dramatic actor in to contend with Will Ferrell being, well, Will Ferrell. But while James Caan, the late, great Misery and The Godfather actor, really sells his performance as Buddy the Elf’s constantly frustrated book publisher father, it turns out he might not have been acting much at all.

“James Caan—may he rest in peace, and we had such a good time working on that movie—he would tease me,” Ferrell said on a recent episode of Christina Applegate and Jamie-Lynn Sigler’s MeSsy podcast (via Entertainment Weekly). “We’d be in between setups, he was like, ‘I don’t get you. You’re not funny. You’re not funny.’ And I’m like, ‘I know! I’m not Robin Williams.’ He’s like, ‘People ask me, “Is he funny?” And I’m like, “No, he’s not funny!” It was all with love.”

While it’s hard to imagine not even cracking a smile at Ferrell’s singing telegram scene, the Anchorman actor said Caan was “truly annoyed” with him throughout production. “I love that the whole time, he’s not acting,” Ferrell shared. “Like, ‘Will this guy shut the fuck up? Jesus!’ So I literally drove him crazy in that movie, just acting like that kid.”

Even if Caan was exasperated every time Ferrell and his yellow tights rolled up to set, it didn’t mean he didn’t care about the movie. In fact, it was his idea to make the boardroom scene—featuring Peter Dinklage, of course—such a dramatic climax. “It was scripted a little more that he would get more frustrated and lose his temper with me, and he didn’t wanna do any of that,” Ferrell explained. “He wanted to save it til that moment in the boardroom where he kicks me out and kicks me out of this life, like, ‘No, it’s gotta be this slow build.’ And he was totally right. He had plotted where his performance was gonna go.”

In the end, Caan finally came around on Ferrell’s particular brand of wackiness. “We were walking out of the theater at the premiere, and we walk out together, and he was like—I take it as like the best compliment, ‘cause he’s coming from James Caan—he’s like, ‘I gotta tell you, I thought everything you were doing while we were filming was way too over-the-top. Now that I see it in the movie, it’s brilliant,’” Ferrell remembered. “That was so funny, he’s walking out, shaking his head, going like, ‘Great job. I thought you were way too over the top. But no, it’s brilliant.’”