No, just the doors designated as emergency exits need to be marked as an exit.  All other doors that appear to be, or could be mistaken as an exit, need to be labeled as “not an exit” or similar designation, or be identified by a sign indicating its actual use (e.g., closet).