Modern telecommunications are moving towards (massive) multi-input multi-output (MIMO) systems in 5th generation (5G) technology, increasing the dimensionality of the systems dramatically. In this paper, the impairments of radio frequency (RF) power amplifiers (PAs) in a 3 x 3 MIMO system are compensated in both the time and the frequency domains. A three-dimensional (3D) time-domain memory polynomial-type model is proposed as an extension of conventional 2D models. Furthermore, a 3D frequency-domain technique is formulated based on the proposed time-domain model to reduce the dimensionality of the model, while preserving the performance in terms of model errors. In the 3D frequency-domain technique, the bandwidth of the system is split into several narrow sub-bands, and the parameters of the model are estimated for each sub-band. This approach requires less computational complexity, and also the procedure of the parameters estimation for each sub-band can be implemented independently. The device-under-test consists of three RF PAs including input and output cross-talk channels. The proposed techniques are evaluated in both behavioral modeling and digital pre-distortion (DPD) perspectives. The experimental results show that the proposed DPD technique can compensate the errors of non-linearity and memory effects in the both time and frequency domains.