Cody, that sounds like an awful lots of primer to me. You don't really need more than one light-to-average coat of primer, especially black. I found black primers go on a lot thicker (and are more "fuzzy"), which can obscure details of the model and makes painting more difficult. It's also more difficult to go dark-to-light than light-to-dark.
In addition, most primers should dry fairly quickly, assuming you are priming in the recommended conditions (which admittedly can be difficult around here). For example, the Dupli-color I mentioned will literally be ready in an hour, assuming you haven't put a ton of primer on. You don't need a lot. Just coverage.
If your primer is still scratching off from normal handling, it's probably because you didn't clean and wash your models before priming. I wash all parts with normal dish soap and a toothbrush post filing mold lines but before assembly. The models are coated in a "dust" of sorts that's used to remove them from the molds after casting. That residue makes the "bond" between the primer and metal less stable.
Clean your models. It's seriously one of the most important steps when building and painting. It will also make the glue bonds stronger when assembling.
Hope that helps!