The origin of the Town of Harvard’s first Town Hall, or Townhouse, is described by Henry S. Nourse in his 1894 History of Harvard:

The earliest movement looking to the building of a hall especially adapted for the transaction of the town’s business was on April 7, 1807, when a committee was appointed to consider the proposition. The report of the committee was probably adverse, as no further action in the matter is recorded, and the town-meetings continued to be held in the meeting-house as they had been from the first. In 1827 the subject was again agitated, perhaps stirred by some natural objections on the part of the first parish to submit their place of worship to the defilement and injury incident to its frequent use by mixed and sometimes disorderly assemblies. A town-meeting debated the question of the town’s right to use the meeting-house, and finally referred it to a special committee for investigation. Samuel Hoar, Esq., was consulted, and advised the town that the edifice was the property of the first parish exclusively, and that a precisely similar case had already been decided by the supreme court in favor of the church in Medford. A for a new building for the town’s use, forty-four by thirty-four feet, estimated to cost seven hundred dollars, but the whole subject was dismissed at that time.

May 5, 1828, a town-meeting was called at the Baptist meeting-house in Still River, and then it was voted to proceed with the erection of a town house at once. The building was placed on the north-eastern portion of the common, across the highway from the present town hall, where E. W. Houghton’s barn now stands. It faced to the south, and had four Tuscan columns supporting the front gable. There was no provision for warming it until 1832, when a chimney was built and a stove purchased.

After a new Town Hall was built in 1871, the old Townhouse was moved slightly to the north (current address 14 Ayer Road) and converted into a residence by George L . Sawyer, who sold it to his father Arad Sawyer. Later in the nineteenth century it was owned by Sawyer’s daughter Sarah and her husband, Charles P. Atherton.