The Gunners took the lead through debutant Marquinhos converting from Eddie Nketiah's cutback after 16 minutes.
Zurich equalised before half-time courtesy of Mirlind Kryeziu's penalty after Nketiah fouled Ola Selnaes.
But Nketiah redeemed Arsenal's lead in the second-half when he headed home at the far post Marquinho's cross.
Mikel Arteta's side face PSV Eindhoven in their second Europa League game next Thursday.
But first they will face a Premier League clash against Everton at the Emirates on Sunday.
