Create a Variable port for all the input port and keep them in a seequence that the variable ports will access first after the input port. Then, compare the same variable ports in the ouput port along with the key columns. If there is any duplicate records found then FLAG it with "X". So that in this way we can eliminate the duplicates from the flat. file.
We can use Aggregate transformation for this. Group by on all the columns in the Aggregator and get only first or last record from the group, it will give you distinct records.
But sorter is the best option, I am not sure why you want to avoid it, is there any specific reason?
Retrieve only Distinct Record from 'flatfile' Source
Questions by nutty.nikhi.l2002