The building was built as a donation from Mrs. Mary B. Young to the people of the city of Fall River, in memory of her son Bradford Matthew Chaloner Durfee, who had died at a young age in 1872, leaving a sizable inheritance.
George A. Clough, a Boston architect was chosen to design the building. Ground was broken in August 1883. The first story of the school is constructed of native Fall River Granite, while the stone of the upper portions is from Mason, New Hampshire. The new high school was formally dedicated to the city on June 15, 1887.
The building occupies a commanding position atop a hill in the city's Highlands neighborhood, and is visible from miles around. Even today, the school's sports teams are known as the "Durfee Hilltoppers".