There isn't a single command in subversion to remove external plugins from your application so I figured I would post this to help the next person save 10 minutes of reading docs.

Make sure you are in the root of your app and do:
svn propedit svn:externals vendor/plugins

Just delete the line(s) of the plugins you no longer want.

If you get the following error then you need to set your environment variable.
svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR is set, and no 'editor-cmd' run-time configuration option was found

You can set your SVN_EDITOR to use TextMate like so:
export SVN_EDITOR='mate -w'