"Solitary Man" was Neil Diamond's first charting single as a recording artist (released in1966). In a 2005 Times Online interview, Diamond said: "After four years of Freudian analysis I realized I had written 'Solitary Man' about myself."

Neil Diamond told Mojo magazine July 2008: "Solitary Man was my first song where I tried to really raise the level of my songwriting. It was inspired by the Beatles' song 'Michelle,' which was also written in a minor key. I don't think I'd ever written a song in a minor key before, it was the first and it kind of broke the dam for me."

Johnny Cash won a Best Country Male Vocal Grammy for his version of the song, which appeared on the 2000 album American III: Solitary Man. The album was produced by Rick Rubin, who produced Diamond's 2008 album Home Before Dark.