In the dark outside our hotel room, I heard a giant blowing over a milk bottle. The hollow, fluted roar was unmistakable. "I think," I said to my sleepy wife, "that there's a whale in the car park."

Here on the Cape in September it was decidedly chilly in the half-light. The sea, 20 metres from the door of our hotel room, had come straight up from Antarctica, bitterly cold but full of microscopic plankton and tiny prawn-like krill, creating a rich seafood soup.

The rocks dropped straight into forty metres of icy black water and right there by the hotel car park were two colossal Southern right whales wallowing quietly in the small, choppy waves. They were so incredibly close that I wanted to get into the water and touch one. But cold and the risk of a great white shark mistaking me for a fur seal kept my feet on the ground. Instead, we stood in quiet awe at the privilege of being fantastically close to these other-worldly giants.

The whales lay side by side, their finless and broad backs slick and smooth. Their heads were encrusted with callosities of barnacles and these patches of white, deep in the water, hinted at the immense bulk dozing below the surface. They were probably 15 metres long, weighing in at around 50 tonnes and easily – by a long way – the largest living animals I had ever been close to.

This might have been a mother and mature calf, a couple of bulls or, perhaps, a romantic assignation. If it was, it wouldn’t be romantic for long. Each year, the whales return from the Antarctic to the shelter of the Cape to calve and breed. Over the last few days, we had seen right whales breaching time and again around the coast, powering their immense, slab-like bodies upwards before slamming down with a shuddering splash.

Because the females only breed every three years, there’s a lot of competition amongst the boys. They jostle, barge and shove. When the female selects a partner, he doesn’t take any chances. Right whales have the world’s biggest balls: 800 kilos that will – he hopes – guarantee that her next calf is his. It puts the whaling phrase “Thar he blows” in a whole new context.

As we stood there, I recalled chatting to an elderly car park attendant in Edinburgh some 20 years ago. As a young man, he said, he was a whaler, butchering Southern right whales by the hundred. And now there I was in a car park in South Africa, dumbstruck by their size and proximity.

A century ago, when Northern right whales had been virtually exterminated, the whalers headed south. International protection was finally conferred on all right whales in 1935 but rogue states, such as the Soviet Union, continued hunting right into the 1960s.

The persecution was intense, with factory ships processing these true monsters of the deep as though they were little more than sardines.

And that analogy with fish is intentional. That afternoon, we headed out on a boat, 35 miles into the south Atlantic. Here, the food-rich Benguela current from the Antarctic mixes with the steamy Alguhas stream coming southwards from the equator and the Indian ocean.

There, we watched a gigantic factory ship reeling in vast nets loaded with tens of thousands of fish. Vast flocks of gannets, petrels and albatrosses wheeled about the ship’s waste chute as it pumped out tonne after tonne of fish guts. Hundreds of fur seals pursued the offal. It was a wildlife extravaganza to be sure, but at colossal environmental cost.

As fish stocks collapse, vast blooms of stinging jellyfish are taking over. Out there in the deep, we are still upsetting the balance. An annual catch of around 300,000 tonnes of krill – a Southern right whale food source – is rising to 750,000 tonnes and new factory ships are being built to catch yet more. Each whale needs to eat up to 2,500 kilos of krill each day and krill also feed fish, penguins, seals and other wildlife. For how long will that be possible?