Schwartz's father, professor of the Leningrad State University, was arrested in 1936 and executed two years later. Schwartz's family was exiled to Kyrgyzstan in 1937, and Schwartz gave private music lessons in Frunze (now Bishkek) as well as occasionally accompanying the silent films at the cinema with live music.
During the Second World War, Schwartz directed one of the sections of the Red Army Choir. During that time, he met Mariya Dmitriyevna, the sister of Russian composer Dmitri Shostakovich, who introduced him to her brother. After the war Shostakovich helped Schwartz gain entry to the Rimsky-Korsakov Conservatory in Leningrad, where he studied composition under Boris Arapov and Orest Yevlakhov. He graduated from the Conservatory in 1951. He joined the Union of Soviet Composers in 1955, soon after the premiere of his Symphony in F Minor.
Schwartz's first major commission was the music for the film Our Correspondent in 1958. He went on to compose the music for more than 120 films, including White Sun of the Desert (1969), The Station Keeper (1972), and The Captivating Star of Happiness (1975). Perhaps his best known work outside of the Soviet Union was for Akira Kurosawa's 1975 film Dersu Uzala.
Schwartz also composed music for three ballets and 35 theatrical performances and, to a lesser extent, for television.
His concerto for orchestra, The Yellow Stars (Purimspiel in a Ghetto), was composed in 2000. It was inspired by the story of the Kovno Ghetto in Lithuania. The concerto was first performed in Saint Petersburg in the same year. In 2002, The Yellow Stars concerto was recorded by the National Philharmonic Orchestra of Russia under the baton of Vladimir Spivakov and in 2005 was released by Capriccio Records on CD and DVD.