You can disable the RSS Feeds folder by modifying the registry:
Outlook 2013
HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\Options\RSS
DWORD: Disable
Value: 1
Outlook 2010
HKEY_CURRENT_USER\Software\Microsoft\Office\14.0\Outlook\Options\RSS
DWORD: Disable
Value: 1
This will just disable RSS functionality, but doesn't remove the RSS folder. Any PSTs created after you enable this registry will not include the RSS folder.
To remove the the RSS folder from your current PST, you have to use the MAPI editor - MFCMAPI Editor: http://mfcmapi.codeplex.com
You should backup your PST then hide the RSS folder (better than delete option) - here you can find a step-by-step tutorial, with screenshots:
http://www.slipstick.com/outlook/delete-outlooks-default-folders/