Lord Fairfax sold some of his five million acres to speculators, farmers and businessmen including Robert Harper and Gersham Keyes.  Harper was a Philadelphia architect who settled in “The Hole” at the confluence of the Shenandoah and Potomac Rivers.  He ran a ferry service across the Potomac from what was then called Shenandoah Falls.  In time, the town at the confluence of the Shenandoah and Potomac Rivers became Harpers Ferry.  Keyes settled on land, which is now within the township of Bolivar and built his home and tavern on Washington Street.  According to tax records for 1790, Keyes owned a grist mill, saw mill, smithy (blacksmith shop), and two distilleries.  He grew wheat and corn, and owned 10 horses, 16 each cows and pigs, 32 sheep and seven pet deer.  Keyes owned eight books, an indication of wealth and education. Keyes also owned two slaves, likely a house servant and skilled worker rather than agricultural slaves.

Sometime just about 1800 the property is sold  to General Harry "Light Horse" Lee, father of Robert E. Lee. "Lee's Legion" was the one of the most successful calavary brigade. He was a delegate to Congress in 1786, and advocated the adoption of the national Constitution in the Virginia convention. He was the ninth governor of Virginia.He was in Congress at the time of Washington's death, and was appointed by that body to deliver the funeral oration at Philadelphia.

Henry "Light-Horse Harry" Lee, served as the Governor of Virginia and earned the respect and friendship of President George Washington as a commander during the American Revolution.

He was a brilliant man but terrible with money. He was involved in one impractical scheme after another. A few of these, such as one for developing land in the vicinity of Washington DC, were merely ahead of their time. Many prominent men of that era were perpetually in debt, owing far more than they could ever repay, and got by only by continually negotiating new loans from their peers.  He was a victim of the land speculation bubble of 1796 causing a crash in the United States real estate market. A series of bad financial investments land him in debtors prison. While in prison, he writes his Memoirs of the War in the Southern Department about the Revolutionary War.

In the 1840s it was known as Graham's Brick Tavern.

The Inn remained intact although derelict until the 1940s. The Light Horse Inn was used by the Ford Foundation as an example of an 18th century inn and tavern. The information gleaned from this building was used to rebuild Williamsburg and several of its Taverns.

In later years the inn was purchased by Amanda Goudie, known by the stage name of Amanda Lane. She was a singer with the Desi Arnaz Band. She had plans to restore the building but her contractor gutted the building and the ran off with her money. She lived for many years with a caved in roof and no exposed beams. Extensive research is ongoing to determine all of the buildings history and layout. There is a large basement summer kitchen with a massive fireplace used for cooking during the tavern days.

Interesting Fact

The county set rates for various commercial enterprises, including tavern and boarding house costs.  If you were to stay at Mr. Keyes “ordinary” or tavern, your costs would be:

Overnight: 7 cents

Breakfast: 10 cents

Supper: 28 cents

Board for your horse: 10 cents

Quart of whiskey: $1.25