In a medium saucepan, bring 4 cups water to a boil over high heat. Remove from heat and add tea bags and rosemary. Let steep for 5 minutes. Remove tea bags. Add sugar and stir until dissolved. Let rosemary steep for 15 minutes more. Remove rosemary sprigs.

Transfer tea to a large pitcher and add remaining 4 cups water and peaches. Cover and let come to room temperature, about 1 hour. Refrigerate and let peaches steep overnight or up to 5 days. Serve over ice. Garnish with rosemary, if desired.