Read This: Alan Rickman's diaries reveal his less-than-flattering opinions about Harry Potter

Severus Snape was always (no Harry Potter pun intended) a controversial figure in the HP fandom. He was a spiteful bully of children and former wizard Nazi, but he also had a big crush on Harry’s mom and dedicated his life to fighting evil in her memory. It’s hard to imagine any actor pulling that off with the kind of gravitas and grace of Alan Rickman.

Rickman documented his journey with Snape—which was apparently almost cut short several times—in his diaries, excerpted ahead of publication by The Guardian. J.K. Rowling famously convinced the late actor to take on the role by (“nervously”) spoiling the Lily love story years before it appeared in the books, something he later described as “a cliff edge to hang on to.”

And yes, he almost fell off that cliff a few times. Before beginning filming Sorcerer’s Stone in 2000, he described “feeling a bit nothing about HP which really disturbs me.” He wrote of decent working relationships with the films’ directors, but can be highly critical of the results: Sorcerer’s Stone, on the big screen, “acquires a scale and depth that matches the hideous score by John Williams,” with the afterparty being “much more fun.” After seeing Half-Blood Prince, he complained of “the need to bang the three Davids’ heads [Harry Potter producers David Heyman and David Barron, and director David Yates] against the nearest wall. I get the character development and the spiffing effects (dazzling), but where is the story????”

His favorite of the bunch seemed to be Prisoner Of Azkaban, praising future Oscar winner Alfonso Cuarón as doing “an extraordinary job.” “It is a very grown-up movie, so full of daring that it made me smile and smile,” he wrote. “Every frame of it is the work of an artist and storyteller.”

He tried to quit several times, including after the second film in 2002 (“Reiterating no more HP. They don’t want to hear it”), and after feeling “shafted” on Goblet Of Fire, but was convinced to stay on to the end (“See it through. It’s your story”). But filming the final films wasn’t smooth sailing: while Snape’s death was “all a bit epic and Japanese,” his last scene with “the Golden Trio” was hampered by what sounds like over-direction from Yates, during which “a small piece of something creative caves in.”

“I found it unsettling to watch,” Rickman wrote of Deathly Hallows Part 2. “[It] has to change horses midstream to tell the Snape story and the camera loses concentration. Audience, however, very happy.”

For more of Rickman’s incandescent opinions—including on his young co-stars (Daniel Radcliffe might not be “really an actor but he will undoubtedly direct/produce” while “Emma [Watson]’s diction is this side of Albania at times”), politics of the day, and his many famous friends and acquaintances (Meryl Streep “turns out to be fun and gossipy. But it’s hard – who else looks like Meryl Streep? So you can’t quite lose the stare”)—check out the excerpt of Madly, Deeply: The Diaries of Alan Rickman in The Guardian.