The Taiwanese world number 48's serve was broken seven times and she saved a match point before winning 3-6 6-4 7-5. "I still think the top players, their average level is higher than, let's say, sub-top players, but anyone on a good day can beat anyone, for sure".
At that point in the match - down 5-2 in the third - it looked as if Hsieh's comeback from a set down would swiftly end.
Karolina Pliskova [No.7] is the only top 10 seed left at SW19 with Serena Williams also still in the running.
"I had to run and fight for every point".
Gulbis, a former world number 10 who famously defeated Roger Federer on his way to the French Open semi-finals in 2014, will be playing in the last 16 at Wimbledon for the first time at the 11th attempt.
In the men's singles, No. 2 seed Rafael Nadal eased into the fourth round as he swept Alex De Minaur 6-1, 6-2, 6-4, continuing his clean record so far.
"This year I have tried to enjoy more, not just tennis but life, food, strawberries and cream, lobster".
Asked when she thought she could win the match, Hsieh responded, "I don't feel I can", then leaned forward and laughed heartily.More news: Henderson hopeful on fitness for World Cup semi-final
And in her third round match against World No 1 Simona Halep at Wimbledon - a tournament that values tradition more than anything else - she was willing to break that service rhythm rule too, at match point no less.
The 21-year-old Swiss will face the victor of German 11th seed Angelique Kerber's match against Japanese 18th seed Naomi Osaka.
A lengthy first game in which Halep finally broke when Hsieh sent a forehand long set the tone in the early stages where holding serve proved hard. "I couldn't stay focused for every ball", Halep said.
Three-time victor Novak Djokovic made the last 16 of a Slam for the 44th time with a 4-6, 6-3, 6-2, 6-4 win over Kyle Edmund, the last remaining Briton.
The Serb, seeded 12th, will meet big-hitting Russian youngster Karen Khachanov for a place in the quarter-final. "I just feel free on the court".
Fifth seed Argentine Juan Martin del Potro has reached the second week of the championships without dropping a set after beating Frenchman Benoit Paire 6-4 7-6 (4) 6-3.
Hsieh has pulled off three recent upsets of top 10 players at Grand Slam tournaments, including a victory against Garbiñe Muguruza in the second round of this year's Australian Open. "She deserved to win". Now she adds a victory over the top player in the world to that list, and does it on the surface that's perhaps most suited to her quirky style of play. Australian Open champion Caroline Wozniacki and defending Wimbledon champion Garbine Muguruza exited in the second, while Madison Keys, Venus Williams and Halep were ousted in the third.
The 2013 doubles champion will now play Dominika Cibulkova after the Slovak beat 15th seed Elise Mertens of Belgium 6-2 6-2.