Consider broken geodesics $\alpha([0,l])$ on a compact Riemannian manifold $(M,g)$ with boundary of dimension $n\geq 3$. The broken geodesics are unions of two geodesics with the property that they have a common end point. Assume that for every broken geodesic $\alpha([0,l])$ starting at and ending to the boundary $\partial M$ we know the starting point and direction $(\alpha(0),\alpha'(0))$, the end point and direction $(\alpha(l),\alpha'(l))$, and the length $l$. We show that this data determines uniquely, up to an isometry, the manifold $(M,g)$. This result has applications in inverse problems on very heterogeneous media for situations where there are many scattering points in the medium, and arises in several applications including geophysics and medical imaging. As an example we consider the inverse problem for the radiative transfer equation (or the linear transport equation) with a nonconstant wave speed. Assuming that the scattering kernel is everywhere positive, we show that the boundary measurements determine the wave speed inside the domain up to an isometry.