I have 38M records in my Source Table. When I move the data from Source to the Target Dimension table, for the very first time (when the Target table is completely empty), takes long time but all 38M records gets Inserted in Target. BTW, I am using my Data Services all along for the Source to Target.
But the problem is with the TYPE 2 Changes. If I make an UPDATE, at Source Table in a comparing column (attribute value), and run the same Data Service, the 'Server Times Out'. Even if I try only for a smaller subset 14M records, (by using 'Where Clause' in the Source 'Query'.)
BTW, If I limit my Source and Target records, 500 or 900 K (Hundred Thousand), my TYPE 2 works just fine..! Any changes in Target gets expired with new record entry, in Target. So, no problem in the Data Service.. Problem is with the large volume of records handling..!
The Failure (Red X) Looks like it is running (Select Statement) again and again, from the 'Target Table. Like this:
Data flow DF_AccountBalance|Reader CachedComparisonTableReader3
SELECT query <SELECT "TCRdr_1"."ACCOUNTBALANCEID" , "TCRdr_1"."ACCOUNTID" , "TCRdr_1"."CENTURY" , "TCRdr_1"."FISCALYEAR"
I see some 'Time Out' message also in here..
timeout expired>. The SQL submitted is <SELECT "TCRdr_1"."ACCOUNTBALANCEID" , "TCRdr_1"."ACCOUNTID" , "TCRdr_1"."CENTURY"
I have Uploaded the entire Data Service Error. Please open in NotePad++ (not in ordinary text reader)
Any thought..?