Duplicate content is content that appears at more than one URL, or is so similar that a search engine treats the pages as the same. Google groups such pages and picks one as the canonical URL to show in the results. According to Google, some duplicate content is normal and does not violate its spam policies.
How does it happen?
Google mentions these causes, among others:
- the same text for different countries in the same language, on separate URLs;
- a separate mobile and desktop version;
- an http and an https version of the site;
- sorting and filtering on category pages;
- accidents, such as a test version of the site that crawlers can reach.
Why does it matter?
There is no penalty, but it does cause problems. Signals such as links get split across several addresses, it is harder to see how a page performs, and Google may pick a different version from the one you want to show.
Example
Say an online shop shows the same jacket at /jackets/rain-jacket and /sale/rain-jacket. A rel="canonical" on both pages pointing to /jackets/rain-jacket tells Google which version you prefer. If the second URL only exists out of habit, a 301 redirect is clearer.
Common mistake
Fixing duplicate pages with a noindex or a robots.txt block. Google advises against both for choosing a preferred version. How to manage filters on category pages is covered in Category pages that get found and help people choose.