It's because the triplets in your 4/4 time section are each 1/3 of a quarter note, whereas the regular eighth notes in your 12/8 time section are longer, being each 1/2 of a quarter note.
In order to play these longer notes at the same speed as the shorter notes in the previous section, you have to speed up the tempo. Otherwise the quarter notes will continue to play at the same speed, and the regular eighth notes will sound slower.
You will see what I mean if you fill some measures with quarter notes in 4/4 and in 12/8 and play them.
Le silence éternel des espaces infinis m'effraie. Church musician