Recent days I am working with a source database that contains Arabic data. If a source is a flat file and a target is template table, we need to set source flat format code page settings to be cp1256 and target table settings enable support nvarchar(if it is MS SQL server) template tables.
But we will face a lot of issues when we are migrating the data from one database to another database with different code page (char set) settings. I have worked recently with MS SQL server, Oracle & Sybase ASE as the source database and Sybase IQ as the target. I have worked the source code pages for oracle are AL32UTF8 and ARISO8859P6, Sybase ASE is UTF8 and MSSQL Server is Arabic_CI_AI_KS etc.
While handling the code page, we need to keep mind the below points else we may face the junk characters issue. Here I am giving example for Arabic data and we can change it other character sets accordingly. SAP Data Services job server code page by default Unicode. No need to change the settings. If you give UTF-8 as server setting it won’t be affected.
- Source database code page and need to set datastore code page settings accordingly.
For Oracle, we need to get code page (char set) using below command
SELECT * FROM NLS_DATABASE_PARAMETERS where parameter = 'NLS_CHARACTERSET'
If Code page is : ARISO8859P6
Language: we can mention the language Arabic etc depending on the code page.
Code page: Database code page iso-8859-6
Server code page: Job server code page. We can keep as default or we can set UTF-8 if we change in the DSconfig file .
If code page is: AL32UTF8
- Target database code page and need to set datastore code page settings accordingly
For Sybase IQ:
Select db_property('Collation')
1256ARA
select db_property('CharSet')
Windows-1256
- Use the environment variables related to databases if required. For example oracle database NLS_LANG environment variable required. For MS SQL Server, Sybase ASE & IQ not required of any environment variables.
NLS_LANG should be sync with OS char set settings and source database code settings if the source is Oracle.
In the above case, AMERICAN is English language and AMERICA is regional setting.
- Set the charset options in the ODBC settings. ODBC settings are depending on the database. For Sybase IQ setting are required but for Oracle , SQL server not required.
- Set the region & language settings in the control panel.
But for the above NLS_LANG settings current system locale should be like below.
- As per the region & language settings in the control panel need to provide the settings in the Data Services Locale selector. Data Services Locale Selector settings written to the DSConfig file. System regional & language settings and DSConfig locale settings should match.
Based on the system local, our data services local selector should be there. Language and territory in the Arabic (Saudi Arabia) in region settings and we gave the same in the in DS settings. Once we give our DSConfig file looks should be like this.
- Restart the system as we are changing the system settings.