A career spanning more than three decades ended on Wednesday last week at London's Harley Street Clinic where he lost his battle with cancer.
The suddenness of his death - he was still taking pictures and playing tennis until a few weeks ago - has stunned the creative community, which saw him as one of a vanishing breed of photographers who learned their craft long before the digital age and strove constantly to produce stylish work.
Wormser will forever be associated with the seminal 1977 B&H ads in which the famous gold pack nestled among the Egyptian pyramids or floated across a swimming pool as a giant sardine can.
It was work that inspired a new style of UK advertising in which art direction took precedence.
"Jimmy was incapable of taking an ugly image," Paul Briginshaw, the Miles Calcraft Briginshaw Duffy creative partner who first worked with Wormser at the then Colman RSCG 15 years ago, recalled.
Adrian Holmes, the chief creative officer at Lowe & Partners Worldwide, said: "He always understood the idea you were trying to convey and how to deliver it in the strongest possible way."
Although he had lived in London for more than 30 years, Wormser never lost his New York accent nor, according to the former agency boss Geoff Howard-Spink, "never failed to give the impression that he'd only just arrived in Britain".
In fact, he knew the arts scene in London better than almost any native, seeking out the best films and art exhibitions and offering forthright opinions on them. His Kensington home was filled with paintings and objets d'art.
It was this love of classic art that he carried with him into his advertising work for clients including British Airways, Martini, Volkswagen, Nike, Guinness and Rolex.
"Jimmy was more than just a photographer," Alfredo Marcantonio, the former agency creative chief, said. "He loved great ideas and didn't like what he saw as an increasing trend for style over substance."
Friends now hope to sustain Wormser's photographic legacy by publishing a book of his work, including shots of the ballet, his abiding passion.
His funeral takes place this Thursday at Mortlake Crematorium at 10.30am.
Anybody who knew him is welcome to attend. A memorial service will be held on 5 November, his birthday.