Kate Gill and Cheryl Salisbury scored in the 86th and 88th minutes, respectively, to tie the game, but Carli Lloyd won it with the game-winner on a header in the crowded penalty area in stoppage time.
The start of the game was delayed by an hour after torrential rains and thunderstorms drenched the field.
The USA will next travel to Birmingham, Ala., to face Australia on Saturday at Legion Field and finish the tour on May 10 against Canada at RFK Stadium in Washington, D.C.
April 27 in Cary, N.C.
USA 3 Australia 2. Goals: Kai 35, Wambach 49, Lloyd 91+; Gill 86, Salisbury 88.
USA -- Solo, Mitts (Buehler, 53), Rampone, Markgraf (Whitehill, 46), Chalupny, O'Reilly (Hucles, 67), Boxx (Tarpley, 68), Lloyd, Heath (White, 46), Kai (Rodriguez, 46), Wambach.
Australia -- William, Reuter (Carroll, 76), Salisbury, Polkinghorne, Colthorpe, Garriock (McShea, 89), Chapman (Gill, 58), McCallum, Shipard (Neilson, 65), Walsh (Kuralay, 82), Simon (Burgess, 59).
Referee: Margaret Domka (USA).