In this study, we used anthropometric data from six Andhra caste populations to examine heritability patterns of 23 anthropometric phenotypes (linear, craniofacial, and soft tissue measures) with special reference to caste differences. We obtained anthropometric data from 342 nuclear families from Brahmin, Reddy, Telaga, Nagara, Ag. Kshatriya, and Mala castes of Visakhapatnam, India. These caste groups represent the existing hierarchical stratification of Indian populations. We used a variance components approach to determine the heritability (h2) of these 23 anthropometric phenotypes (height, weight, BMI, etc.). The sample consisted of 1918 individuals ranging in age from 6 to 72 years (mean = 21.5, S.D. = 13.8). The heritabilities (h2 ± S.E.) for all anthropometric traits for the entire sample were significant (p < 0.0001) and varied from 0.25 ± 0.05 (BMI) to 0.61 ± 0.05 (bizygomatic breadth) after accounting for sex, age, and caste effects. Since data on socioeconomic and nutritional covariates were available for a subset of families, we repeated the genetic analyses using this subset, which has yielded higher heritabilities ranging from 0.21 ± 0.16 (head breadth) to 0.72 ± 0.18 (nasal breadth). In general, craniofacial measurements exhibited higher h2 compared to linear measures. Breadth measurements and circumferences yielded more or less similar heritabilities. Age and sex effects were significant (p < 0.0001) for most of the traits, while the effects of caste, socioeconomic status, and nutritional status were inconsistent across the traits. In conclusion, anthropometric phenotypes examined in this study are under appreciable additive genetic influences.