In present study we exploited Langmuir technique to produce self-assembled arrays composed of monodisperse iron oxide nanoparticles 10 nm and 20 nm in diameter and of their binary mixture. A combination of in-situ X-ray reflectometry and Grazing-incident small-Angle X-ray scattering was used to obtain in-plane and out-of-plane structure of the arrays directly on the water surface. Surface pressure isotherms and X-ray reflectometry analysis showed that monodisperse 10 nm nanoparticles form a highly ordered monolayer, while 20 nm particles pack in three-dimensional clusters with a short-range (nearest-neighbor) correlations between the particles. In a binary mixture of 10 nm and 20 nm nano particles composed in proportion 3:1 the self-assembly process results in a structure where the monolayer of 10 nm particles is perturbed by the larger particles. Non-trivial mixing causes an enlargement of interparticle distance but keeps the symmetry of two-dimensional lattice of smaller nanoparticles. Estimation of the acting interactions and micromagnetic simulation suggest the optimal formation for monodisperse and binary ensembles.