An increasing number of studies have compared climate change adaptation policies within and between different countries. In this paper we show that these comparative studies suffer from what is known as the “dependent variable problem’ – the indistinctness of the phenomenon that is being measured, and disagreement on its scope and boundaries. This problem has been signaled in other scientific fields where it proved to hamper meaningful comparisons and policy evaluations, transnational learning, and policy transfer. This paper aims to raise consciousness of the dependent variable problem in comparative studies on climate change adaptation policy by exploring its origins and proposes ways to deal with it. Three main sources of the problem are discussed: (1) conceptual indistinctness of adaptation policy and the heterogeneity and lack of consistency of what is being compared between cases. (2) Inadequate research designs to compare cases. (3) Unclear indicators and explanatory variables to compare across cases. We propose a way to operationalize the concept of adaptation policy, provide a narrower description of the research designs for policy change or outcomes analysis, and finally discuss possible measurements concepts.