C# Remove Duplicate Lines From Text File

There are various methods to remove duplicate lines from text file. Some of these are easier to implement than others. The approach to be taken can depend on the size of the text file and the expected number of matching lines. Here we focus on small size text files.

String[] TextFileLines = File.ReadAllLines("textFile.txt");

TextFileLines = TextFileLines.Distinct().ToArray() ;

File.WriteAllLines("textfile.txt", );

In the C# code above, first we read all text file lines and stored these lines into an array. Then we used Distinct() function, a built in function of C#, to remove duplicates. Finally we wrote these distinct lines back to the file.