The 'correct' solution is to apply the right amount of oil (which is a judgement call based on experience, but having spoken to many customers, it's always less than they think. A typical bottle of oil lasts a very long time, I'm about halfway through only my second bottle for all the bats I have knocked in since we opened) and also to allow adequate time for it to soak in. Typically this is only a day or so, but if you've over oiled it can take longer. I have occasionally had bats that have taken 3 or 4 days to soak up their oil to the point where they will take tape/anti scuff sheets, but these are a rarity.
The quick fix, which isn't really going to cause any problems other than maybe some very minor scratching to your bat is to give the bat a going over with very fine sandpaper.