About 1835, house designer and builder Robert Cutler constructed the house at 81 Lessey Street in Amherst for Luke Sweetser. As described in The History of the Town of Amherst, Massachusetts, Vol. I (1896):

Leonard Dwight and H. Wright Strong conducted a general store in Amherst prior to 1812; in April of that year, the partnership was dissolved and a new one formed by Strong and Elijah Dwight. H. Wright Strong was in business in 1815; his store stood on the site now occupied by Adams’ drug-store. In 1824, Luke Sweetser, who had been for three years in Mr. Strong’s employ, bought out the business, continuing it under his own name until March, 1830, when his brother, J. A. Sweetser, became associated with him under the firm name of L. & J. A. Sweetser. A few years later, J. A. Sweetser severed his connection with the firm and William Cutler and Luke Sweetser formed a partnership under the firm name of Sweetser & Cutler. In 1848, George Cutler became a member of the concern, the name being changed to Sweetser, Cutler & Co. Mr. Sweetser retired from the business in 1857, and the Cutler brothers continued it under the firm name of W. & G. Cutler. The firm of Geo. Cutler & Co. was formed in 1870 and continued in business until 1884, when it was succeeded by the present firm of Jackson & Cutler.

Sweetser, who after his retirement from business focused on farming, died in 1882 and his house became the Oak Grove School for girls. After 1903, it was the Phi Gamma Delta fraternity house and was remodeled by architect Karl S. Putnam in 1929 with a monumental columned Greek Revival portico. When Amherst College abolished fraternities in 1984, the house became a dormitory called Marsh House, named for Eli Marsh, a professor of Physical Education and member of Phi Gamma Delta.