For being 4 inches onto property which you were not using anyways? I don't know what the law would say, but I think that since you had no prior agreement with the property owner about the cost of the fence or what sort of fence I don't think he should have to pay for half of it. The removal of his attachment may cause more damage then good with regards to your own fence and your relations with that neighbor.
First, he should have asked your permission. What happens if you take down your fence? Will his collapse? Bad planning on his part.
Second, you'd have to contact your city planners or other appropriate agency for the laws. The only specific fencing laws I found for Missouri were about fences and livestock.
Third, IMO I don't think he should pay for half the fence, but I do think financial remuneration of some sort would be in order.