\(y''-4y=0\)
has characteristic equation
\(r^2-4=0\)
which has roots at \(r=\pm2\), giving the characteristic solution
\(y_c=C_1e^{2x}+C_2e^{-2x}\)
For the nonhomogeneous part of the ODE, let \(y_p=(a_2x^2+a_1x+a_0)\sin2x+(b_2x^2+b_1x+b_0)\cos2x\). Then
\({y_p}''=(-4b_2x^2+(8a_2-b_1)x+4a_1-4b_0+2b_2)\cos2x+(-4a_2x^2+(-4a_1-8b_2)x-4a_0+2a_2-4b_1)\sin2x\)
Substituting into the ODE gives
\((-8b_2x^2+(8a_2-b_1)x+4a_1-8b_0+2b_2)\cos2x+(-8a_2x^2+(-8a_1-8b_2)x-8a_0+2a_2-4b_1)\sin2x=(x^2-3)\sin2x\)
It follows that
\(\begin{cases}-8b_2=0\\8a_2-8b_1=0\\4a_1-8b_0+2b_2=0\\-8a_2=1\\-8a_1-8b_2=0\\-8a_0+2a_2-4b_1=-3\end{cases}\implies\begin{cases}a_2=-\dfrac18\\\\a_1=0\\\\a_0=\dfrac{13}{32}\\\\b_2=0\\\\b_1=-\dfrac18\\\\b_0=0\end{cases}\)
which yields the particular solution
\(y_p=-\dfrac18x^2\sin2x+\dfrac{13}{32}\sin2x-\dfrac18x\cos2x\)
So the general solution is
\(y=y_c+y_p\)
\(y=C_1e^{2x}+C_2e^{-2x}-\dfrac18x^2\sin2x+\dfrac{13}{32}\sin2x-\dfrac18x\cos2x\)
