Interesting problem! You don't mention the rocket size, or whether you hand ram or press. That would be helpful to know. Also, what kind of clay bulkhead did you use, and how thick was it? What did the top surface of it look like? Pics? Do you use a tube support?
You say you are using raw ball-milled BP, and also mention using the finest airfloat. I'm assuming that means you are ball-milling the complete 60-30-10 mixture. That's not typically done. 'Standard' BP rockets use finely powdered or milled ingredients, but not milled together like BP. A typical nozzled BP coreburner would use half of the 30 parts of charcoal as airfloat, and the other half would be coarser, like 36 mesh or 80 mesh. The charcoal would be made from slower burning hardwoods, not hot BP charcoals. That's what I do, and I have no problem at all with blowthroughs. You mention using the powder with no moisture in it. I use a couple of percent of water, spritzed and screened in. This helps greatly with consolidation. The clay is never moistened. Most folks don't use water in the propellant, but it's becoming more popular in my mind.
The way to "weaken the reactivity" of the BP won't be needed if you do as I mentioned above. These aren't my ideas, they are generally accepted by most.
Since we don't know the size of rocket, we can't be sure that your hope to lift a 5 inch shell is possible, but it's doubtful (in my opinion). A 1lb nozzleless rocket with hot 75-15-10 will lift a 4 inch ball shell to proper height every day of the week. 5 inch, not so much. My guess is that your 60-30-10 ball-milled propellant in a nozzleless configuration won't be enough for a 4 inch- in the 1lb size. If you were to use 60 milled potassium nitrate, 15 fine charcoal, 15 coarser charcoal, and 10 sulfur with a nozzle you should be able to comfortably lift a 3 inch shell, maybe a 4 inch. Some of us use a hot BP charcoal for the 15 parts of fine charcoal instead of commercial airfloat to get more zip.
You should try lifting dummy shells with kitty litter and burst before you take a chance with a live shell.
Weingart's advice was good, but: he suggests using "mixed coal" in his propellant, not a ball-milled complete mixture.