Today I found " table comparison " is very weird , so that I don't know how could I use it well .
My etl job is very simple .
Extract data from flatfile, compare these data with one table(in fact, this table is empty) to find new and updated records ,and insert into another table finally .
I use " sorted input " in "table comparison", and I know I need order these data before " table comparison".
And I use two columns as "input primary key column" .
Below are part of my data in flatfile .
2009.01.11 00:00:00|1.3638|Y|EUR@USD@39814@Y|EUR|2009.01.01 00:00:00|USD
2009.01.11 00:00:00|10.157|Y|GBP@CNY@39814@Y|GBP|2009.01.01 00:00:00|CNY
2009.01.11 00:00:00|1.0917|Y|GBP@EUR@39814@Y|GBP|2009.01.01 00:00:00|EUR
2009.01.11 00:00:00|137.09|Y|GBP@JPY@39814@Y|GBP|2009.01.01 00:00:00|JPY
2009.01.11 00:00:00|1.489|Y|GBP@USD@39814@Y|GBP|2009.01.01 00:00:00|USD
There're 4500 records in flatfile , and 7 columns per row. And one of columns contains 3 "@" .
But this job hang . I haven't saw it end .
I replaced "@" using "#" or "%" . This job also hang.
But the weird thing is that , if I replace "@" using "(" or "!" or "-" , this job run very quick ( less than 5 seconds).
And if I reduce the rows of flatfile to 900 rows, this job also run quick.
Because of this problem, I am very sad and I have a thought to not use bods.
Could someone met the same problem ?