The Young House, at 1 Fairbank Street in Harvard, was built in the early nineteenth century, although the exact date is unknown. In 1825, the property was sold by John P. Whitcomb to cordwainer William Lewis, who sold it to James Young in 1836. Young then transferred it to Dr. Samuel Young , who was probably his uncle. As described in the History of the Town of Harvard (1894), Vol. 2, by Henry Stedman Nourse, Dr. Young was

born in Athol August 12, 1782, son of Lt. Samuel and Lois (Sanderson) Young. Dr. Young was a graduate of Williams College, 1804, and practiced in Athol and Lowell before coming to Harvard. He lived for about thirty years in a house yet standing upon the east side of the common, where he died March 30, 1845. One of his legs being much shorter than the other, he walked with a cane. He was the last of the old-style doctors, paying his visits on horseback, his stock of medicines borne in saddlebags before him.

In the mid-nineteenth century the house was owned by his daughter Seraphina and her husband Hiram Joy and was called Joy Cottage. The house passed among female descendents until 1985.