Olivier grabbed took five wickets for 51 as South Africa gained a first innings lead of 77 at the Wanderers Stadium.
But the hosts were unable to forge a position of dominance, struggling to 135 for five at the close, an overall lead of 212.
Hashim Amla made a patient 42 not out off 90 balls against tight, disciplined bowling, while Quinton de Kock lifted the scoring rate late in the day by scoring 34 not out off 35 balls.
The wickets were shared between four bowlers with medium-pacer Faheem Ashraf taking two for 19 — with both his wickets coming in his first over — to reduce South Africa to 45 for four.
Fast bowler Olivier was twice involved in crucial breakthroughs as Pakistan were bowled out for 185 in their first innings.
He ended an hour of frustration for South Africa when he took two wickets in four balls immediately after the morning drinks break, ending the stubborn resistance of nightwatchman Mohammad Abbas, who batted for 88 minutes and faced 51 balls in scoring 11.
Three balls later Asad Shafiq ducked a bouncer but was caught behind off his glove.
Abbas and opening batsman Imam-ul-Haq, who went on to make 43, were both dropped twice during the first hour of play. South Africa also missed two run-out opportunities.
Pakistan were 91 for five when Imam was caught at second slip off Vernon Philander but Babar Azam (49) and captain Sarfraz Ahmed (50) put on 78 off only 61 balls for the sixth wicket.
Kagiso Rabada ended the stand when Sarfraz was caught at first slip and four balls later Olivier had Azam caught at long leg. Olivier made it a double strike when Faheem Ashraf was out first ball, splicing a mistimed pull to short leg.
Olivier started the series as South Africa’s fifth-choice fast bowler but played in the first Test in Centurion because of injuries to Vernon Philander and Lungi Ngidi. He had match figures of 11 for 96, prompting South Africa to opt for a four-man pace attack when Philander returned for the second Test.
South African stand-in captain Dean Elgar was caught behind for five off Mohammad Amir and first innings top-scorer Aiden Markram followed after making a breezy 21.
The South African innings was in disarray when Ashraf dismissed both Theunis de Bruyn and new cap Zubayr Hamza in the space of two balls before Amla and Temba Bavuma (23) added 48 for the fifth wicket. It was left to De Kock to lift the tempo.