Preheat oven to 400 degrees. With a fork, prick the skins of 2 peaches, 2 plums, and 2 nectarines. Halve each fruit, and remove pits. Halve fruit again.
In a roasting pan, toss fruit with 2 sprigs fresh rosemary, 2 to 3 tablespoons sugar (depending on sweetness of fruit), 2 tablespoons unsalted butter, cut into pieces, and 1 tablespoon fresh lime juice. Cook, tossing occasionally, until fruit is fork-tender, 15 to 20 minutes.
Serve warm over vanilla ice cream or frozen yogurt, if desired, and drizzle pan juices over the top.