In 1741, William Havens and his wife were expecting their first child. On property purchased in 1700 by his grandfather George Havens from Nathanial Sylvester 2nd , William built a modest house.  James duly arrived on February 12, 1742. The building  of Havens House was completed by 1743.

On his father’s death in 1763, James, then 21, assumed management of the homestead. He had married Elizabeth Bowditch and was at that time the father of two sons and a daughter. (James and Elizabeth had eleven children). By 1769 James opened a store in this house, and expanded his activities to include a post office and a small school. Store it partly was, as ledgers attest, but a major item coming in on ships from the Caribbean was rum. Archival ledgers show James often sold the rum in bulk,  but there are many ledger entries that show purchases of liquor consumed on the premises as “a mug of flip” or a “nip of grog.”  The house functioned as a tavern and is so designated in records.

Into this household burst the Revolution! In May of 1775, 43 Shelter Island Patriots (including James Havens) signed a resolution of support for the Continental Congress. On June 25, Islanders James Havens, Thomas Dering and Daniel Brown were elected to the NY Provincial Congress of 1775-1776.
In August of 1776 the battle of Long Island was fought and lost, and Shelter Island became occupied territory along with the rest of Long Island. General Washington recommended Island families take refuge in Connecticut, and James took his family to safety there.

During the first part of the war, James was a member of the refugee militia at Fort Griswold in New London, CT. He was later a Captain in the Suffolk County, NY militia. Many Islanders were engaged at sea in the privateer and transport service. Under orders from General Washington, James Havens became commander of the Privateer Brig Jay. At the end of the hostilities, he returned to Shelter Island with his family, to find it had been ravaged. Fortunately, Heartsease was intact, with only a few things stolen by the British troops.

Havens kin remaining on the Island kept the farm partly secure from enemy raiding and the store staggered on with meager supplies and greatly reduced patronage. The Island was plagued with constant raids from enemy ships in Gardiner’s Bay, and in 1781 marauding crews carried off from the house valuables which included a watch, a coat, and a fowling piece, along with foodstuff.

At the close of the hostilities in 1783 the family had all returned, to find their island ravaged but their home intact. Between 1783 and 1785, James was the Town Supervisor and the house served as the meeting place of the annual Town Meeting. The store and tavern once again became a thriving gathering place, as the island settled in and life returned to normal.

The Long Island Herald, published in Sag Harbor, reports in 1792 that
“On Feb. 22, the birthday of President Washington was celebrated
on Shelter Island at the house of Captain James Havens.” 

It is clear that Captain James was well liked and the family must have led a pleasant life in the old house, which he  named “Heartsease.