csvkitHow to remove duplicates with csvkit?
csvkit is a suite of command-line tools for working with CSV data. It can be used to remove duplicates from a CSV file.
Example code
csvdedupe input.csv output.csv
Output example
Processing input.csv
Writing output.csv
The code above will take the input file input.csv
and remove any duplicate rows, writing the output to output.csv
.
Code explanation
csvdedupe
: the command to run the deduplication processinput.csv
: the input file containing the data to be deduplicatedoutput.csv
: the output file where the deduplicated data will be written
Helpful links
More of Csvkit
- How to convert a tsv file to csv with csvkit?
- How to update specific records using csvkit?
- How to fetch unique values with csvkit?
- How to split a csv file with csvkit?
- How to convert a csv file to tab delimited with csvkit?
- How to select specific columns with csvkit?
- How to merge columns with csvkit?
- How to reorder columns with csvkit?
- How to skip the first line with csvkit?
See more codes...