It's usually due to lack of water, even with a lack of bees there's usually enough insects to at least pollinate some of the flowers. If you've got a mist attachment for the hosepipe try giving them a good misting as it is said to help setting.
We grow ours in 3'x6"x6" troughs (6 of them) in a mixture of compost and farmyard manure and so far we've had 4 feeds and put a couple of pounds of them in the freezer.