Hi haydenareadb,
You can accomplish this using the Import / Export Tool, which you can access from Customize > Integration Tools in Method.
First you'll want to export your Contacts table (uncheck the Show only common tables? checkbox), exporting the fields FullName, RecordID and TagList. This will generate a CSV file that you can open up in Excel. Do that, making the changes you want (go through and remove your tag(s) from the appropriate contacts) and saving the file somewhere on your computer.
Next, go back to the Import / Export Tool and import your modified CSV file back into the Contacts table. You'll want to make sure to select Update existing records by matching the key fields and select RecordID as the key field on Step 2 of the import.
You can use the same process to add tags to multiple contacts at a time, just add your tag to the TagList field rather than removing it. Just keep in mind that in this situation, you'd have to go in and create the tag manually beforehand. You can only use the Import / Export tool to apply tags to your contacts if those tags already exist.
We have a Help Center article on how to use the Import / Export Tool in case you get stuck, you can find it here. :)
Hope this helps.
- Justin