Zlatko Dalic's side, which reached the World Cup semifinals last year, moved a step closer to earning its first ever trophy and demonstrated once again it's overtime specialists.
Andrej Kramaric and Mario Pasalic netted in the second half after Donyell Malen sent the Dutch ahead, but Noa Lang snatched a 96th-minute equalizer to force an additional half hour.
Petkovic got away from Frenkie de Jong, however, and drilled home from outside the box to decisively put Croatia ahead eight minutes into overtime, and Modric wrapped up the win with a penalty.
The Nations League represents one of the last chances for Croatia's elder statesmen, including Modric, to taste international triumph.
"It's a victory for the Croatian people, we beat the Netherlands in front of their fans, Croatia has one more medal -- it's impossible, unreal, but she deserves it," Dalic told his country's Nova TV. "Nothing upset the team, they know what they are playing for, above all, Croatia. You endure everything for Croatia. This is one of our biggest victories, to win 4-2 on their ground, we were losing, we conceded in the 96th minute. Surely, this is a victory which will go down in history."
Croatia has only suffered one defeat in its last 16 matches, to World Cup champion Argentina, and quashed the Netherlands' bid to win its first competition since the 1988 European Championship.
Ronald Koeman's team finished as runner-up in the inaugural Nations League against hosts Portugal in 2019, while France triumphed in 2021.
"We fought like lions, but unfortunately it was just not good enough today," Netherlands captain Virgil van Dijk told Dutch broadcaster NOS. "It's very disappointing. We knew it would be difficult, but we actually did quite well. We conceded two penalties, we have to be better there, it's hard to look for words now."
Koeman selected Malen on the right of the attack, with exciting youngster Xavi Simons making his fourth appearance for the Netherlands on the left. Both were involved in the opening goal, along with Liverpool forward Cody Gakpo, as the Oranje burst into life after half an hour.
The match started sluggishly in the evening heat, both sets of players tired after a long season with the World Cup wedged in the middle.
After a tidy team move Feyenoord's Mats Wieffer, playing in his club's De Kuip stadium, fed Malen, who slotted past Dominik Livakovic -- Croatia's World Cup hero on its run to third in Qatar.
Bouncing back. Croatia came out stronger in the second half, with Kramaric cutting inside from the left and firing wide of the far post.
Soon it was level, when 37-year-old Modric, earning his 165th Croatia cap, cleverly robbed Gakpo on the edge of the area, with the forward pulling him back to concede a penalty.
Kramaric stroked his spot kick down the middle as Justin Bijlow dived to his right to delight Croatia's traveling contingent.
Further celebrations were in order when Luka Ivanusec crossed for Pasalic, the Atalanta midfielder, to stab home from close range after 73 minutes.
Dutch defender Nathan Ake, who featured in Manchester City's Champions League final win just five days ago, came close with an acrobatic effort before Lang pounced on a loose ball to level.
Croatia, which has beaten Brazil, Japan, England, Russia and Denmark in matches that have gone to overtime at the last two World Cups, was nonplussed.
The Netherlands could not maintain parity for long, with Petkovic beating Bijlow at his near post.
Lang fired into the side netting when he might have equalized and Netherlands paid the price when Tyrell Malacia fouled Petkovic in the area and Modric converted the penalty to seal the win.
Croatia, third in the 1998 and 2022 World Cups and runner-up to France in 2018, has its sights firmly set on their first silverware.
"We have bronze and silver, let's go for gold so that we can finish this story," said Dalic.
• Spain and Italy face off in Enschede on Thursday in the other semifinal.
© Agence France-Presse