In the paper we present detailed analyses of two machine translation systems when applied to documents of a previously unseen domain: agricultural texts from the European Union. The two systems compared are a statistical machine translation (SMT) system using the freely available ISI ReWrite Decoder (Germann 2003a), and the rule-based machine translation system MATS (Sågvall Hein et al. 2002). For the purpose of evaluation we use a sentence-aligned Swedish-English corpus of approximately 75,000 words per language, where 90% are used for training and 10% are used for evaluation. In the paper we discuss the outcome of automatic evaluation and the results of our manual quality assessment.