Heat the oven to 375 F.
Wash the potatoes and pierce with a fork. Place in the oven and bake until the skins are crisp, about 1 hour.
Carefully — potatoes will be very hot — cut the potatoes in half and scoop out the pulp, leaving about 1/8 inch of the potato flesh attached to the skin. Save the pulp for another use.
Spray the inside of each potato skin with butter-flavored cooking spray. Press in the rosemary and pepper. Return the skins to the oven for 5 to 10 minutes. Serve immediately.