The beguine is a type of dance. “When they begin the beguine” is a song by Cole Porter written in 1935, in which the singer recounts how and when he hears the music for that dance, it brings back memories. So sometimes the phrase is used as a reference to the song.

When they begin the beguine
It brings back the sound of music so tender,
It brings back a night of tropical splendor,
It brings back a memory ever green.

I'm with you once more under the stars,
And down by the shore an orchestra's playing
And even the palms seem to be swaying
When they begin the beguine.

To live it again is past all endeavor,
Except when that tune clutches my heart,
And there we are, swearing to love forever,
And promising never, never to part.

What moments divine, what rapture serene,
Till clouds came along to disperse the joys we had tasted,
And now when I hear people curse the chance that was wasted,
I know but too well what they mean;

So don't let them begin the beguine
Let the love that was once a fire remain an ember;
Let it sleep like the dead desire I only remember
When they begin the beguine.

Oh yes, let them begin the beguine, make them play
Till the stars that were there before remain above you,
Till you whisper to me once more,
"Darling, I love you!"
And we suddenly know, what heaven we're in,
When they begin the beguine