This paper presents entropy symmetrization and high-order accurate entropy stable schemes for the relativistic magnetohydrodynamic (RMHD) equations. It is shown that the conservative RMHD equations are not symmetrizable and do not admit a thermodynamic entropy pair. To address this issue, a symmetrizable RMHD system, equipped with a convex thermodynamic entropy pair, is first proposed by adding a source term into the equations, providing an analogue to the nonrelativistic Godunov--Powell system. Arbitrarily high-order accurate entropy stable finite difference schemes are developed on Cartesian meshes based on the symmetrizable RMHD system. The crucial ingredients of these schemes include (i) affordable explicit entropy conservative fluxes which are technically derived through carefully selected parameter variables, (ii) a special high-order discretization of the source term in the symmetrizable RMHD system, and (iii) suitable high-order dissipative operators based on essentially nonoscillatory reconstruction to ensure the entropy stability. Several numerical tests demonstrate the accuracy and robustness of the proposed entropy stable schemes.