pythonRemove duplicate lines from text file
We'll remove duplicate lines from input.txt and write result to output.txt
lines_seen = set()
with open('output.txt', 'w') as output_file:
for each_line in open('input.txt', 'r'):
if each_line not in lines_seen:
output_file.write(each_line)
lines_seen.add(each_line)ctrl + clines_seendeclare set to store unique lines | with openopens specified file for reads |
if each_line not in lines_seencheck if this line is duplicate | output_file.writewrite unique line to |
lines_seen.addsave unique line to seen lines | |