Datras Dynamic ABAP in Datras EPTL

Export of Industrial Amount of Data from SAP

General

Extracting data via the SAP function RFC_READ_TABLE has some limits, which are not acceptable in a professional, resp. industrial environment.

Major problem

SAP RFC interface does not support a persistent database cursor. That means, after each RFC call or RFC call-back all database cursors are closed. Consequently, all data have to be exported by a single RFC call, which yields to an inacceptable load on the SAP server and on the network considering industrial amounts of data.

Poor man's solution

Fragmentation of the data export by using the parameter ROWCOUNT and ROWSKIPS of the SAP function RFC_READ_TABLE cannot solve this problem, because this does only shift the problem from the SAP server to the database server:

1. Because SAP does not support scrollable database cursors, all following RFC calls have to skip over the extracted data before. That means, the network traffic between the database server and the SAP server will increase by N * (N-1) / 2, whereby N represents the count of packets. Increasing server and network load in dependence on the square of transferred packets is inacceptable.
2. Furthermore, using this solution, it is not possible to guarantee a consistent image of data, because all changes done between the successive RFC calls will be reflected in the exported data.

An other approach

We do not extract and transfer the data by one RFC call, but we first store the data to be exported by one single RFC call in a temporary file on the SAP sever and then transfer this file in small packets from the SAP server to the client by multiple RFC calls. That means, we replace the missing persistent database cursor of SAP by an always existing file pointer of the operating system.

This solution has the following advantages:

1. We only need to keep one data record in memory, because we store the data record immediately after fetching the record from the database server. This decreases significantly the load on the SAP server.
2. We transfer the data exported via RFC in many small packets, so we can decrease network load significantly. Furthermore, using this method, we can introduce rock and roll principle, that means, we split calls of RFC_CALLRECEIVE into separate calls of RFC_CALL and RFC_RECEIVE, which allows an asynchronous work of the server and the client.
3. We have a consistent data image, because all data are extracted by only one single call.
4. Using file streaming, we can serialize the exported data. So, we can kill the 512 byte limit of the SAP function RFC_READ_TABLE.

Unfortunately, this approach does not solve the problem with the missing SQL elements like nested selects, aggregates and joins. If we want to solve this problem, we need to replace the predefined select statement of the SAP function RFC_READ_TABLE by a hardcoded select statement. Therefore, the client has to provide a dynamically generated ABAP including a hardcoded select statement and using the same mechanism to export the selected data as discussed above.

Minor problems

1. SAP text conversion

SAP text conversion of non-text type as used by the SAP function RFC_READ_TABLE is not very sophisticated:

• SAP text conversion of data type FLTP is totally useless.
• SAP text conversion of data type DEC, CURR and QUAN can yield to an overflow resulting in a leading asterisk of the string, because SAP mixed OUTPUTLEN and INTLENG of table DD03L (found by Thomas Roth, CAS).
• It is simply not possible to convert the full range of a signed INT4 into an array of ten chars!

Therefore, we store non-text data types in binary form as hexadecimal chars and do the data type conversion by our own in the client.

2. SAP dynamic ABAP

SAP only supports dynamic type casting from release 6.10 and up. However, there exists a problem by dynamically type casting INT1, because data type INT1 can be mapped to both ABAP types B and X (see table DD03L). Therefore, we better forswear this new feature, and have to use the asterisk in the select clause. This has the additional advantage to be completely independent of the SAP version.

3. SAP Unicode

It seems, that when writing data into a binary stream, SAP only converts data of text type internally from UTF8 to UTF16, but stores the other data types as Ansi chars.
In contrast, when reading data from a binary stream, all data are internally converted from UTF16 to UTF8 and the half bytes of non-text data are dropped.
Therefore, we first map data type INT1, INT2, INT4, DATS, TIMS, FLTP, DEC, CURR and QUAN into a char array.

4. Little-Big-Endian problem

We cannot resolve the little-big-endian problem in ABAP, because SAP does not support type casting INT2 and INT1 to type I. This problem has to be resolved by the client in detecting the current hardware of the SAP system by testing well-known values.

5. SAP align bug

Defining variables used for type casting deeper in the body of the ABAP source can yield to an align error followed by a crash of the SAP interpreter. Therefore, we define all variables, which are used for type casting, at the top of the ABAP source.

6. SAP function RS_FUNCTION_DELETE

Calling the SAP function RS_FUNCTION_DELETE can yield to zombies in the ABAP workbench, because the SAP table DIRTREE is not correctly updated. Therefore, we update the SAP table DIRTREE manually after using the SAP function RS_FUNCTION _DELETE when removing ABAPs (caught and fixed by Thomas Roth, CAS).

7. SAP zombies

It seems to be a general problem that zombies remain in SAP when installing and removing ABAPs via RFC. Therefore, we perform a client-side reconnect to SAP after installing and removing ABAPs via RFC.

Unsolved problems

Due to lack of data SAP data types ACCP, LCHR, LRAW, VARC, STRG, RSTR could not be tested consequently. If there appears any problem, please contact the vendor.

Acknowledgments

Many thanks to Thomas Roth, CAS, for help when fixing SAP Unicode problems.

Many thanks to Nord-IT for the patience and nerves when implementing the Unicode interface.

© Datras 2010