The Baxter-Patrick library is located near the southwest corner of James Island, on a site with a documented history stretching back more than 350 years. The library stands near the east side of the Stono River, which takes its name from the Stono Indians who occupied this area seasonally during the English settlement of the Carolina colony in the 1670s. The Stono soon clashed with Europeans and their desire to control the landscape, however, so the tribe formally ceded their traditional territory in 1684 and moved southward. From that moment commenced a long succession of planters who farmed several hundred acres of land here using the labor of enslaved people of African descent.
The land surrounding the library site was once used to grow a variety of crops, including corn, sweet potatoes, indigo, rice, and Sea Island cotton. In addition, the enslaved people living here raised dairy cows, beef cattle, hogs, and sheep. Paul Chaplin Grimball acquired this site in 1833 and expanded it into a plantation containing over 1,000 acres farmed by more than one hundred enslaved laborers. The Grimball plantation was abandoned during much of the Civil War, however, when Union soldiers arriving from the Stono River occupied this site on several occasions and skirmished nearby with Confederates forces.
After the war, much of the old plantation was divided into rental farms leased to formerly enslaved people and their descendants. Grimball heirs continued planting a variety of crops here, although on a much smaller scale. The arrival of the boll weevil ended the cultivation of Sea Island cotton in the 1920s, but the construction of South Grimball Road and Folly Road in the 1930s facilitated truck farming of vegetables to inland markets. In the 1950s, Grimball heirs subdivided and sold much of the old plantation, including lots surrounding an old African-American cemetery located near what was formerly the center of the old plantation.
Charleston County purchased this site in 1963 and built Baxter-Patrick Elementary School, named for two retired school principals, Mrs. Nan C. Baxter (died 1966) and Mrs. Anna S. Patrick (died 1973), who are remembered as the first black school teachers on James Island during the era of segregated education. That building was demolished in 2011, and in June 2015, Charleston County selected the site of the former school as the home for a state-of-the art public library for the people of James Island. Construction commenced in early 2018, and the Baxter-Patrick James Island Library, containing 20,000 square feet of community resources, opened to the public on November 2nd, 2019.