When we use only Aggregator transformation in our mapping for approx 5 milion records it takes 40-42min time but when we use it with a sorter transformation, the time reduces to 12 -13 min. We have also noticed that throughput of select statement from source was also very high. It aggregates grouped data quickly is ok but why throughput of select statement was also higher using sorter transformation?

  • Interview Candidate
  • 2
  • 4287

Showing Answers 1 - 2 of 2 Answers

sai

  • Feb 14th, 2007
 

passing data to sorter is time taking why because sorter transf stores whole data in buffer and then sorts based in specified ports.insted of passing data to sorter transf ,sort d data in the sourse qualifier transf by overriding the sql query in the properties tab.it will take less time to do the job . i hope u got the answer.

  Was this answer useful?  Yes

priyank_divya

  • Jul 11th, 2012
 

When an Aggregator transformation is used without sorter, it stores all the data before performing the grouping operation. But when a sorter is used before aggregator, it sorts all the data before passing it to the Aggregator transformation.When the source records pass through the aggregator transformation, it groups the rows based on the group by function once the rows for that coulmn(used in group by) are passed to it.

eg.

eno ename
1 A
2 B
1 C

Once the record " 2 B" is passed to the aggregator t/f, it groups the records for "1" which is not the case when sorter is not used.

  Was this answer useful?  Yes

Give your answer:

If you think the above answer is not correct, Please select a reason and add your answer below.

 

Related Answered Questions

 

Related Open Questions