Southern U.S. derogatory term for "poor, white trash" (1766), is from mid-15c. crack "to boast" (e.g. not what it's cracked up to be), originally a Scottish word. Especially of Georgians by 1808, though often extended to residents of northern Florida.
Cracker referred to the white man who used the whip. They were called crackers because that is what they were doing with the whip.