The 19-year-old, unwanted at Derby County, fired the winner for the League Two side five minutes from the end.
Palace boss Peter Taylor made nine changes to the side which played at Birmingham last Saturday, although skipper Marcus Hughes netted to put them ahead on 15 minutes.
However the Eagles rarely raised their game so it came as no surprise when Lawrie Dudfield levelled seven minutes later.
In not a particularly exciting match, County will have lots to celebrate after Martin, a Welsh Under-21 international, capped a great performance with an unstoppable first time volley from 12 yards out.
In fact Martin had a part to play in Dudfield's goal with a well measured pass which the Magpies hit-man stuck beautifully from the edge of the box past the Palace keeper Scott Flinders.
It was the perfect response from Steve Thompson's outfit after Hughes put Palace in front with another cracking goal, this time finishing off Clinton Morrison's pass by rifling a rising left-foot shot into the roof of the net.
With so many changes, the Eagles looked disjointed and only in a reasonable early spell in the second half did they show signs of breaking down the Magpies. In this period Morrison headed wide and Ben Watson hit the side-netting but that was it from the home side.
Meanwhile, Tcham N'Toya-Zoa was always a danger and was unlucky with a second-half shot which flew a fraction wide of the post.