During Port’s heyday, early 1800’s, the "Turner Ashby House", as it was often called by locals, was built for Benjamin Franklin Kemper, son of Dr. George Whitfield Kemper, Sr. In addition to practicing medicine, Dr. Kemper was an entrepreneur, investing in a number of businesses: mercantile store, woolen mill, foundry, etc. One son, Dr. G.W. Kemper, Jr., also practiced medicine in Port, while Frank (Benjamin Franklin) oversaw family business enterprises.

The home of Benjamin Franklin Kemper and his wife, Eliza Holbrook Kemper, was built as one room over one with a side hall in the 1830's, but the rest of it, the larger part, was added within 10 years. Their residence also operated as an inn and tavern for travelers, primarily for river boatmen.

When Port Republic became the site of a major battle during "Stonewall" Jackson's Valley Campaign, 1862, the Kemper house was used as a "hospital"; General Jackson was headquartered in the upper end of the village at Madison Hall, home of Dr. Kemper, Sr. On June 6, the famed Confederate General Turner Ashby was killed in battle near Harrisonburg. His body was brought behind the battle lines to the Frank Kemper House, explaining why it is often referred to as the "Turner Ashby House".

Legend holds that the women of Port prepared his body for burial, placing a red rose over the wound where the bullet had entered. The body, wrapped in a Confederate flag, was laid in the front corner room of the house, under a window. General Jackson was among those who came to pay respects. According to Wilse Harper, a twelve-year-old at the time, Jackson patted him on the head and said, "Son, there lies the best General a man ever had".

After the Civil War, the property changed hands often, first purchased by J.T.L. Preston. Around the turn of the century the house was operated as a "tea room", at another time, a boarding house. One lady recalls that she and her husband, who was an N & W Railroad Agent, had rooms in this house, then owned by Charles F. Saufley, from May through November in 1935.

When the house was put on the market in 1991, the Society of Port Republic Preservationists saw an opportunity to assure the preservation of this historic structure and its access to the public, while also affording an excellent facility for a museum/visitor center, a major goal of the Society. An intensive fund-raising campaign during the following year resulted in its purchase in June, 1992. Civil War enthusiasts, history lovers, and caring individuals joined to provide essential funds without any moneys from public sources or foundations.