Performant extraction of most recent data from a history table

You have a table that contains multiple time stamped records for a given primary key:

Output required is the most recent record for every key value:

Solution #1: Use the gen_row_num_by_group function

Build a dataflow as such:

In the first query transform, sort the input stream according to Key and Timestamp desc(ending). The sort will be pushed to the underlying database, which is often good for performance.

In the second query transform, add a column Seqno and map it to gen_row_num_by_group(Key).

In the third query transform, add a where-clause Seqno = 1 (and don’t map the Seqno column).

Solution #2: use a join

Suppose we’re talking Big Data here, there are millions of records in the source table. On HANA. Obviously. Although the sort is pushed down to the database, the built-in function is not. Therefore every single record has to be pulled into DS memory; and then eventually written back to the database.

Now consider this approach:

The first query transform selects two columns from the source table only: Key and Timestamp. Define a group by on Key and set the mapping for Timestamp to max(Timestamp).

In the second transform, (inner) join on Key and Timestamp and map all columns from the source table to the output.

If you uncheck bulk loading of the target table, you’ll notice that the full sql (read and write) will be pushed to the underlying database. And your job will run so much faster!

Note: This second approach produces correct results only if there are no duplicate most recent timestamps within a given primary key.