Since you have the same symptom on both batteries, I would not exclude the charger.
If you have a way of "tapping" into the output while charging you may find the charger is not putting out sufficient voltage. Even on dead batteries you should see the output to be higher than battery voltage while charging.
If you can attempt to charge the batteries with an external source (other fully charged battery), as has been suggested, you then can determine if it's the charger or the batteries.
When you stored the drill and batteries, were they fully charged...?