The Cape Ann Museum in Gloucester purchased the Babson-Alling House in February 2019 and the historic home is currently undergoing restoration to become part of a new museum campus that will include a new storage and programming facility, slated to open in June 2020. Located at the modern address of 243/245 Washington Street, the house was built in 1740 by Joseph Allen (1681-1750) or his son, William Allen (1717-1815), at what was then the town center, called the Green, adjacent to the White-Ellery House, also owned by the museum. In 1765, William Allen sold the house to Isaac Smith (1719-1797), a wealthy Boston merchant and slave owner who was the uncle of Abigail Adams. Scipio Dalton was an enslaved person who is thought to have lived in the attic of the house. He eventually gained his freedom from Smith in 1783 after a period of indenture.

Smith sold the house in 1779 to John Low, Jr. (1754-1801), a merchant and lieutenant in the militia. The house passed to his daughter, Eliza Gorham Low (1786-1862), who married Nathaniel Babson (1784-1836), a merchant and ship captain, in 1809. The house was eventually inherited by their son Gustavus Babson (1820-1897). Most of his brothers became seafarers, but Gustavus was a successful farmer on the property. He married his first cousin, Susan Stanwood Low (1820-1880). Their daughter, Ann Prentiss Babson Alling, moved to the house after the death of her husband in 1894 and maintained the property with her brother Nathaniel. Her daughter, Elizabeth L. Alling, also lived in the house for many years. The house may have been a stop on the Underground Railroad before the Civil War.