Container gardens and worms

In the past I've usually planted veggies/herbs directly into meter square raised planter boxes, but the last couple of years due to a boisterous digging puppy I haven't planted anything! All 3 empty raised garden boxes have had sheep pellets, blood and bone and grass clippings piled into them for 2 years and we now have lovely crumbly soil with HUNDREDS of worms.
This time what I want to do is use one of the boxes and place 16 x 12L plastic square buckets inside and plant in the containers. I'm trying to see if the dog is still an idiot, plus just feel like experimenting after being inspired by http://www.hungrybin.co.nz/customers/large-scale-processing/ [scroll down to see the fabulous waist high planters}!
Question is. can I use the crumbly soil that we have. WITH the worms [and more sheep pellets etc] The containers will have drainage holes drilled in them, will be inside with the box with tops level with top of planter. Will the worms be happy and survive merrily or will I cause them to stress out and die?
[probably a dumb question with obvious answer. but I want to play it safe] ;-)
TIA

. or should I pick the worms out and just use the soil

looks like I'll avoid the worms!
http://tropicalhorticulture.blogspot.co.nz/2011/02/earthworms-kill-potted-plants.html

You can use the soil with the worms as long as the worms have a way to escape the pot when they run out of food. However, the conventional thing to do when using worm soil is to remove the worms first and use them to populate the next worm bin, sell the excess worms, feed them to fish/chooks/etc. Worms will nibble on the plant roots if they run out of food and can't get out.

Thanks.
I've piled all the wormy soil into another garden area and have decided to go with a veggie potting mix to be on safe side.
Also decided to plant one of the 3 boxes with veggies as normal and see which one fares best. Naughty dogs just don't help :-)