Fatty acid binding protein 4 (FABP4) is an abundant protein in adipocytes, and its production is influenced by high-fat diet (HFD) or obesity. The prostate stromal microenvironment induces proinflammatory cytokine production, which is key for the development and progression of prostate cancer (PCa). Here, we show that high FABP4 expression and its secretion by PCa cells directly stimulated PCa cell invasiveness by upregulating matrix metalloproteinases through phosphatidylinositol 3-kinase and mitogen-activated protein kinase signaling pathways. In addition, prostate stromal cells augmented PCa cell invasiveness by secreting interleukin-8 and -6 in response to FABP4. This was abrogated by the FABP4 specific inhibitor, BMS309403. Furthermore, a mouse xenograft experiment showed HFD enhanced PCa metastasis and invasiveness by the upregulation of FABP4 and interleukin-8. Clinically, the serum level of FABP4 was significantly associated with an aggressive type of PCa rather than obesity. Taken together, FABP4 may enhance PCa progression and invasiveness by upregulating matrix metalloproteinases and cytokine production in the PCa stromal microenvironment, especially under HFD or obesity.