The compassionate use of available medications with unproven efficacy is often in conflict with their clinical evaluation in placebo-controlled clinical trials. For ultra-rare diseases where no approved treatments exist, such as fibrodysplasia ossificans progressiva (FOP), routine clinical trial enrollment for available medications may be difficult to achieve. Therefore adaptive methods of evaluation are often desirable. Off-on-off-on (O4) approaches offer an opportunity to rapidly assess the potential symptomatic efficacy and tolerability of a medication with a limited number of patients and may aid in the design of more focused clinical trials that are amenable to enrollment. Here we report three children with classic FOP who had recalcitrant flare-ups of the back and who had been treated with an O4 regimen of imatinib. In all three children, fewer flare-ups, decreased swelling and improved function with activities of daily living were reported by the parents and treating physician when the children were “on” imatinib than when they were “off” imatinib. The median time to improvement on imatinib was 2–3 weeks. The anecdotal O4 experience with imatinib reported here in three children with FOP who had recalcitrant flare-ups of the back supports the design of a brief placebo controlled trial to assess the potential efficacy of imatinib in reducing the symptoms in children with refractory flare-ups of FOP. A tool to prospectively measure and quantitate flare-up symptoms is presently being developed and validated and will be used for such a study.