Settled in 1772 and named after Zachariah Spencer, a loyalist who became a patriot during the Revolutionary War. In 1874, J. Harvey Wilson II built a water-powered textile mill (Wilson & Moore Cotton Mill) at the base of Spencer's Mountain. W. Thomas Love and John C. Rankin purchased the mill in 1895 and changed its name to Spencer Mountain Mills. A predecessor of Duke Power purchased the mill and the hydroelectric plant that powered it in 1926. The mill was purchased by Pharr Yarns in 1957 and was shut down in 1999.
Spencer Mountain is located at the foothills of a mountainous ridge.