Caster wheel bolt stuck in the fork

So I am pretty sure the bearing/bushing inside my Gravely caster wheels have been shot for some time. Loose etc. Never affected the cut so I didn't bother fixing them.

Then this week one of them locked up and dragged on a lawn. Time to fix.

So the nut came off easily an the bolt will spin in the fork but for the life of me I can't get the bolt out of the fork. It is not threaded- turning it does not get it out. And even hitting it with a hammer won't make it budge. Am I missing...

Caster wheel bolt stuck in the fork