Explosive increase in the smart phone users and mobile services have resulted in dramatic increases in wireless data traffic. In order to meet the increasing user demand, the mobile communication industry is moving rapidly towards 4G cellular systems often called long-term evolution (LTE and LTE-Advanced). The leading carriers and vendors have launched the LTE services to provide improved service quality over 3G systems in terms of throughput, spectral efficiency, latency, and peak data rate. Among many technologies adopted in 4G cellular systems, multiple-input multiple-output (MIMO) technology has played a key role in achieving these diverse goals. In a short period, various forms of MIMO technologies, including single user MIMO (SU-MIMO), multiuser MIMO (MU-MIMO), and coordinated multipoint transmission and reception (CoMP) have been proposed in LTE and LTE-Advanced (LTE-A) standard. In particular, multiuser MIMO (MU-MIMO), in which the base station transmits multiple streams to multiple users, has received much attention for achieving improvement in performance. From the initial release (Rel. 8) to the recent release (Rel. 10) so called LTE-Advanced, MU-MIMO techniques have been evolved from its premature form to the more elaborated version in a short period of time. In the first part of dissertation, we provide an overview of design challenges and the specific solutions for MU-MIMO systems developed in LTE-Advanced standard. In the second part, we study multi-cell downlink systems which have been actively investigated in literatures. Specifically, we divide the multi-cell downlink strategy into two classes: i) interference-aware precoding and ii) network MIMO with limited backhaul. In the former case, eNBs located close to each other cooperate and jointly design their transmission strategies with exchange of only channel state information (CSI). In other words, the eNBs do not exchange the messages intended for UEs. In the last part, for the network MIMO systems, the eNBs are connected to a common central unit which has all messages intended for the UEs located in the geographical region covered by the connected eNBs. For the interference-aware precoding systems, we study the advantage of multi-layer broadcast coding at the eNBs which enables a sophisticated multi-user decoding at the UEs. For the network MIMO systems, we assume that the backhaul links connecting the eNBs and the central unit have finite capacity. We propose a generalized joint design of backhaul and radio access communication systems. For both interference-aware precoding and network MIMO systems, we formulate the weighted sum-rate maximization problem subject to per-eNB power constraints (for the latter, we also have additional backhaul capacity constraints) under the assumption of Gaussian single-user coding of UE messages. It will be shown that all the formualted problems belong to the class of difference-of-convex problems and we will derive iterative algorithms converging to locally optimal solutions.