There are 2 ways of adding custom nameservers - the proper way or the dirty way...
The dirty way is just allowing other nameservers to share the same IP addresses as your primary nameservers, if you allow this via WHM then all you need to do is setup the nameservers at the registrar - you can allow the sharing of the nameserver IP's via WHM > Server Config > Tweak Settings and then create the reseller account, then add the nameserver entries in the DNS zone for that domain
The correct way of doing it is to use seperate IP addresses for these custom nameservers, go to the reseller center, select the reseller account and press edit priveledges / nameservers - at the bottom you can input the nameservers so that all accounts created by that reseller use the new custom nameservers and not the standard main nameservers... make sure you press the "Add an A entry for this nameserver" after each entry to setup the DNS, the system will use any free system IP addresses for the nameservers...
Read here with screenshots
http://www.webhostgear.com/71.html