I guess this would depend on the kind of responsibility: legal? no, not as individuals.
Moral? yes, I think so.
The thing is, racism concerns us all! a society where some people are discriminated will not thrive and even those that are not discriminated will suffer indirectly (by higher taxes, crime rate, etc),. Preventing racism is in the best interest of everyone!
