1. As was suggested by @sideburns did a package-cleanup --cleandupes to get rid of the fc21 duplicates.
2. Ran dnf --releasever=22 distro-sync to install the few hundred packages which were left out of the upgrade transaction.
3. System was still booting with the last fc21 installed kernel, so updated grub and everything seemed to work fine:
grub2-mkconfig -o /boot/grub2/grub.cfg