Politically, Astaire was a conservative and a lifelong Republican Party supporter. With Bing Crosby, George Murphy, Ginger Rogers and others he was a charter (founding) member of the Hollywood Republican Committee.
Always immaculately turned out, Astaire remained something of a male fashion icon even into his later years, eschewing his trademark top hat, white tie and tails (which he never really cared for) in favor of a breezy casual style of tailored sports jackets, colored shirts, cravats and slacks — the latter usually held up by the idiosyncratic use of an old tie in place of a belt.
Astaire married for the first time in 1933, to the 25-year-old Phyllis Potter (née Phyllis Livingston Baker, 1908–54), a Boston-born New York socialite and former wife of Eliphalet Nott Potter III (1906–81), after pursuing her ardently for roughly two years. Phyllis's death from lung cancer, at the age of 46, ended 21 years of a blissful marriage and left Astaire devastated. Astaire attempted to drop out of the film Daddy Long Legs (1955), offering to pay the production costs to date, but was persuaded to stay.
In addition to Phyllis Potter's son, Eliphalet IV, known as Peter, the Astaires had two children. Fred, Jr. (born 1936) appeared with his father in the movie Midas Run, but became a charter pilot and rancher instead of an actor. Ava Astaire McKenzie (born 1942) remains actively involved in promoting her late father's heritage.
His friend David Niven described him as "a pixie — timid, always warm-hearted, with a penchant for schoolboy jokes." Astaire was a lifelong golf and Thoroughbred horse racing enthusiast. In 1946 his horse Triplicate won the prestigious Hollywood Gold Cup and San Juan Capistrano Handicap. He remained physically active well into his eighties. At age seventy-eight, he broke his left wrist while riding his grandson's skateboard.
He remarried in 1980, to Robyn Smith, an actress turned jockey almost 45 years his junior. Smith was a jockey for Alfred G. Vanderbilt II.
Astaire has never been portrayed on film. He always refused permission for such portrayals, saying, "However much they offer me – and offers come in all the time – I shall not sell." Astaire's will included a clause requesting that no such portrayal ever take place; he commented, "It is there because I have no particular desire to have my life misinterpreted, which it would be."