Response to the Referees’ report on the paper ”Study protocol for the evaluation of a vocational rehabilitation”

Response to the Referees’ report on the paper ”Study protocol for the evaluation of a vocational rehabilitation” Philip Fowler, Xavier de Luna, Per Johansson, Petra Ornstein, Sofia Bill and Peje Bengtsson. We thank the reviewers for their constructive feedback and suggestions for revision as well as for the speedy review process. We have revised the paper as described below. Reviewers’ comments are reproduced in italics, followed by our answers. As part of the revision, some errors were found in our R code. We have corrected these and updated all the corresponding tables and figures. The results are very similar to the previous ones. We apologise for this inconvenience. Reviewer 1 The stratification strategy for waiting time seems arbitrary. Were the cutoffs based on quantiles? (Doesn’t appear to be based upon sample sizes in table 4) If not, how were they chosen? Table 4 shows both the treated and the controls in each of the strata. The stratification strategy was based upon the deciles of waiting times for the treated (that fulfilled our inclusion criteria). Using those quantiles would give the following cutpoints: [31, 70), [70, 90), [90, 109), [109, 126), [126, 146), [146, 166), [166, 187), [187, 214), [214, 249) and [249, 361). However, some of these strata do not respect the fact that the regulations for sick leave change by Swedish law once an individual exceeds a waiting time of 90, 180 or 365 days (Socialförsäkringsbalk, 2010, 27 kap. 47-49 §). Since the regulations might change the probability of being treated as well as affect the outcome, the decile-based cutpoints were corrected to take this into account. We now clarify this in the paper on page 4, lines 39-42. Are subjects aware of their prognosis? If so, assumption 3i seems very strong to me. A longer prognosis (whether or not the prognosis is appropriate) might be abused by a subject who could really return to work earlier (e.g. had a speedier then average recovery). This is an important question. The subjects were not aware of their prognosis. Furthermore, the caseworkers were informed that the prognosis variable would not be used to evaluate their performance as caseworkers. We clarify this in the paper on page 5, lines 22-25. Algorithm steps 2 and 3 - The discarding strategy in these two steps are at odds. Step 3 proceeds to restrict the number of discarded treatments, but step 2 discards at the arbitrary cutoff of needing at least 5 controls. Why not be more flexible with that threshold? We agree that it would be possible to have a variable number of matches (5 or less) within a given caliper instead of discarding treated not having five matches. While perhaps more appealing 1 than simply discarding the observations for which five controls could not be found, it has not been implemented in the Matching package used to perform our analysis. We do not expect that a change in implementation would make much of a practical difference however. In Step 3, we need to put a restriction on how many treated observations are allowed to be deleted to make sure that best balance is not obtained by simply deleting a large number of treated in Step 2. Reviewer 2 Assignment to the treatment group. There is limited information provided concerning how individuals are assigned to the JAM tx group, and no information for the post-match activities of the control group. A richer description of both would allow for a better assessment of the appropriateness of the proposed research design. Caseworkers are trained to make decisions regarding eligibility to get access to sickness benefits. This training is a priority at the SIA and the process is monitored on a yearly basis, ending in a report of how to make improvements, see, e.g., Swedish Social Insurance Agency (2016). Thus while the decision of JAM assessment is at each caseworker’s discretion, and thus might vary between caseworkers, it is expected to be based on similar principles. This is now clarified in the paper in the first paragraph of Section 2.3.1. Also, see our response to the next comment for information about the post-match activities of...