The First Presbyterian Church downtown in Tallahassee with it's slave galley upstairs and it's gun ports in the basement from the days where the curch was on the perimiter of the city and the citizens sought protection from Indian raids. First Presbyterian Church (established 1832) in Tallahassee, Florida is the oldest church in the city and one of the oldest churches in the state was built in 1833 when Florida was still a territory..