This article offers a reading of the colonialist and ableist (audist) assumptions that undergird Daniel Defoe’s Robinson Crusoe (1719). I argue that Crusoe’s fraught experience of sound on the island serves as punishment for sins he committed in his youth (indicating the novel’s audism) and that his use of gesture allows him both to manage his auditory uncertainty and to con solidate power over Friday. Through my analysis of John Bulwer’s universal theory of gesture, I situate that cultural history alongside the emergence of deaf education in the eighteenth century. Reading gesture as critical to the colonial philosophy of Robinson Crusoe provides insight into travel narratives by Christopher Columbus and William Dampier, who convey that sign was used in the absence of interpreters in contact zones. Such representations of gesture make legible colonial designs and Indigenous resistance. Gesture destabilizes Crusoe’s authority even while it enables him to establish his rule. My reading of Robinson Crusoe argues that colonialism and audism work together to justify Western European expansion.