Airpods Pro 3 go mute on Mac after a few minutes (no disconnect)

I am on a Macbook Pro M3 (26.0.1) and have Airpods Pro 3 (8A358).

The Airpods connect and play sound. After a few minutes they will suddenly stop playing sound.

There is no ui indication or disconnect happening the sound is juste gone. Sometime I can still change the noise mode settings and will hear the confirmation sound. On some occasions after changing the mode the sound will come back.

A simple reconnect fixes the issue temporarily.

My Airpods Pro 2 don't have the issue paired with the same mac.

I tried to "forget device" and reconnect but the issue remains.