Categories are a way of grouping products that have shared characteristics. You can have subcategories inside of main categories. An example of this might be the 3 categories "Tools", "Hardware", and "Clothing". Inside of the "Tools" category you could have 3 subcategories "Power Tools", "Hand Tools", and "Automotive".
Tags are a simpler way of showing products that relate to one another. You can't nest tags inside of each other like you can for categories. An example of a item tag could be "New Item" or "Featured Item"
Attributes are a way of assigning items custom properties that allow for searching and sorting. A store manager can make unique attributes that provide additional information about the products. Examples of custom attributes are "Size", "Color",