Elizabeth Margaret Baillie (1794–1876) and William Hunter Baillie (1797–1894), were the children of Dr Matthew Baillie, the renowned pathological anatomist. Little is known about the life of Elizabeth – she was a companion of Walter Scott's daughter, Sophia.

Considerably more is known about William. He trained as a barrister but never practiced, choosing instead the life of a gentleman of leisure. He was close to his aunt, Joanna Baillie, and shared her interest in the history of the Hunter-Baillie family. He assembled the family's papers, including correspondence, ancient title deeds and other legal instruments, in order to establish the pedigree of the family. William also encouraged his aunt to write her memoirs. William and his wife Henrietta had eight children, five of whom, three sons and two daughters, died before William did, the other son surviving for just three months after William’s death.