-
How to improve the performance while reading data through sequentilal file?
tapan8984
- Aug 12th, 2011
We can use the Number of readers per Node property to enhance parallelism in Sequential file, hence enhancing the performance..
-
-
How to configure ODBC drivers to work with Data stage ?
SinhaS
- Aug 4th, 2011
1) First check that the Driver has been installed and the library env variable has been set correctly to point to it by the UNIX admin who installed it ($LD_LIBRARY_PATH). 2) Go to $DSHOME and update ...
-
What is base partitioning
What is base partitioning in DataStage how we can implement
-
Ways of implementing scd2
How many number of ways that you can implement SCD2 ? Explain them
-
Datastage Questions
what is multiple instances?how u trigger the job in windows?how can u see,delete dataset in unix and windows?what is the function of shared container?why don't we use odd nodes to run the job?what is the normal view and materialized view?when do u use separate sort stage and inbuilt sort utility in a stage?what is the difference?if a job is aborted?can v run the job with out compilation?how do u commit...
j.padma89
- Jul 16th, 2011
How to check the no. of nodes while running the job in UNIX? Answer : type in the directory path where the apt_config_file is located,then u can see the no of nodes and other config information like ...
j.padma89
- Jul 16th, 2011
If a job is aborted?can we run the job with out compilation
Answer:For any condition ("abort " or "reject"), one should run a job only after compilation (strictly) -
Job Endtime
How to find job endtime?
-
How to run 10 jobs at the same time in DataStage 8.0.1?
j.padma89
- Jul 14th, 2011
To run 10 or any number of jobs one after the other or in any sequence, One can use "job sequences".
In each sequence one can assign a job, so after the 10 sequences , you can run a sequence job.
Using sequential jobs developer can even decide what should a job do if it is aborted.
amulas
- Jul 13th, 2011
By using multiple job compile from ds director, we can run multiple jobs at a time.....
-
Incremental loading
How to design a job with incremental loading in datastage?
-
What is the difference between Datastage Server jobs and Datastage Parallel jobs?
saratgunji
- Jul 13th, 2011
Both Server jobs and Parallel jobs run on Data Stage server only. where are server jobs performance is slower than parallel jobs because parallel jobs run on SMP and MPP. parallel jobs has partition...
nikhilanshuman
- Nov 29th, 2009
Datastage parallel jobs can run in parallel on multiple nodes. Server jobs do not run on multiple node.Parallel jobs support partition parallelism(Round robin,Hash,modulus etc.), server jobs don...
-
How do we pass parameters from UNIX?
How do we pass parameters from UNIX? & How do we pass parameters by using UNIX shell scripting?
mcrao1
- Jul 11th, 2011
Hi Amulaswe can pass parameter in UNIX command by using '-PARAM'.syntax:???? -PARAM parameter_name %parameter value%Example:??? -PARAM business_date %20110711%??????RegardsChalapathirao.MHCL Technologies( Malaysia)+60182296096.
-
How can we filter in sequential stage
dhora9999
- May 23rd, 2011
Ya you are right, If you do not know the answer please do not cut paste here. Please answer the questions promptly. do not confuse anybody..Thanks D!!
rajani
- Feb 21st, 2007
In the Filter option,u can use the UNIX command like this to filter the input records..head -3 input_file.dat
-
Improve performance using copy stage
How can we improve performance using copy stage
Datastage Etl
- May 13th, 2011
Copy stage is used to send the data to the multiple sources.?Here we can ?change the column names . Rather than using Transformer Stage where ever necessary , we can use the copy stage. Like this, c...
narra satish
- Mar 21st, 2011
By using copy stage we can increase the performance. In this stage we can do sorting, Removing unwanted columns
-
How did u connect with DB2 in your last project?
Most of the times the data was sent to us in the form of flat files. The data is dumped and sent to us. In some cases were we need to connect to DB2 for look-ups as an instance then we used ODBC drivers to connect to DB2 (or) DB2-UDB depending the situation and availability. Certainly DB2-UDB is better in terms of performance as you know the native drivers are always better than ODBC drivers. 'iSeries...
Read Best Answer
Editorial / Best Answer
nikhilanshuman- Member Since Nov-2009 | Dec 17th, 2010
By making use of "DB2/UDB Stage".There we basically need to specify connection parameters like client instance name,default server,default database.
nikhilanshuman
- Dec 17th, 2010
By making use of "DB2/UDB Stage".There we basically need to specify connection parameters like client instance name,default server,default database.
-
What are other Performance tunings you have done in your last project to increase the performance of slowly running jobs?
Staged the data coming from ODBC/OCI/DB2UDB stages or any database on the server using Hash/Sequential files for optimum performance also for data recovery in case job aborts.Tuned the OCI stage for 'Array Size' and 'Rows per Transaction' numerical values for faster inserts, updates and selects.Tuned the 'Project Tunables' in Administrator for better performance.Used sorted data for Aggregator.Sorted...
Read Best Answer
Editorial / Best Answer
sistlasatish- Member Since Nov-2005 | Nov 14th, 2005
- Minimise the usage of Transformer (Instead of this use Copy, modify, Filter, Row Generator)
- Use SQL Code while extracting the data
- Handle the nulls
- Minimise the warnings
- Reduce the number of lookups in a job design
- Use not more than 20stages in a job
- Use IPC stage between two passive stages Reduces processing time
- Drop indexes before data loading and recreate after loading data into tables
- Gen\'ll we cannot avoid no of lookups if our requirements to do lookups compulsory.
- There is no limit for no of stages like 20 or 30 but we can break the job into small jobs then we use dataset Stages to store the data.
- IPC Stage that is provided in Server Jobs not in Parallel Jobs
- Check the write cache of Hash file. If the same hash file is used for Look up and as well as target, disable this Option.
- If the hash file is used only for lookup then \"enable Preload to memory\". This will improve the performance. Also, check the order of execution of the routines.
- Don\'t use more than 7 lookups in the same transformer; introduce new transformers if it exceeds 7 lookups.
- Use Preload to memory option in the hash file output.
- Use Write to cache in the hash file input.
- Write into the error tables only after all the transformer stages.
- Reduce the width of the input record - remove the columns that you would not use.
- Cache the hash files you are reading from and writting into. Make sure your cache is big enough to hold the hash files.
- Use ANALYZE.FILE or HASH.HELP to determine the optimal settings for your hash files.
This would also minimize overflow on the hash file.
- If possible, break the input into multiple threads and run multiple instances of the job.
- Staged the data coming from ODBC/OCI/DB2UDB stages or any database on the server using Hash/Sequential files for optimum performance also for data recovery in case job aborts.
- Tuned the OCI stage for 'Array Size' and 'Rows per Transaction' numerical values for faster inserts, updates and selects.
- Tuned the 'Project Tunables' in Administrator for better performance.
- Used sorted data for Aggregator.
- Sorted the data as much as possible in DB and reduced the use of DS-Sort for better performance of jobs
- Removed the data not used from the source as early as possible in the job.
- Worked with DB-admin to create appropriate Indexes on tables for better performance of DS queries
- Converted some of the complex joins/business in DS to Stored Procedures on DS for faster execution of the jobs.
- If an input file has an excessive number of rows and can be split-up then use standard logic to run jobs in parallel.
- Before writing a routine or a transform, make sure that there is not the functionality required in one of the standard routines supplied in the sdk or ds utilities categories.
Constraints are generally CPU intensive and take a significant amount of time to process. This may be the case if the constraint calls routines or external macros but if it is inline code then the overhead will be minimal. - Try to have the constraints in the 'Selection' criteria of the jobs itself. This will eliminate the unnecessary records even getting in before joins are made.
- Tuning should occur on a job-by-job basis.
- Use the power of DBMS.
- Try not to use a sort stage when you can use an ORDER BY clause in the database.
- Using a constraint to filter a record set is much slower than performing a SELECT ? WHERE?.
- Make every attempt to use the bulk loader for your particular database. Bulk loaders are generally faster than using ODBC or OLE.
sistlasatish
- Nov 14th, 2005
Minimise the usage of Transformer (Instead of this use Copy, modify, Filter, Row Generator)Use SQL Code while extracting the dataHandle the nullsMinimise the warningsReduce the number of lookups in a ...
-
What are OConv () and Iconv () functions and where are they used?
IConv() - Converts a string to an internal storage formatOConv() - Converts an expression to an output format.
Read Best Answer
Editorial / Best Answer
sekr- Member Since Dec-2005 | Jan 30th, 2006
iconv is used to convert the date into into internal format i.e only datastage can understand
example :- date comming in mm/dd/yyyy format
datasatge will conver this ur date into some number like :- 740
u can use this 740 in derive in ur own format by using oconv.
suppose u want to change mm/dd/yyyy to dd/mm/yyyy
now u will use iconv and oconv.
ocnv(iconv(datecommingfromi/pstring,SOMEXYZ(seein help which is iconvformat),defineoconvformat))
sekr
- Jan 30th, 2006
iconv is used to convert the date into into internal format i.e only datastage can understand example :- date comming in mm/dd/yyyy format datasatge will conver this ur date into some number like :- 7...
-
How did you connect to DB2 in your last project?
Read Best Answer
Editorial / Best Answer
gagan8877- Member Since May-2007 | May 8th, 2007
The following stages can connect to DB2 Database:
ODBC
DB2 Plug-in Stage
Dynamic Relational Stagegagan8877
- May 8th, 2007
The following stages can connect to DB2 Database:ODBCDB2 Plug-in StageDynamic Relational Stage
-
What is Metastage?
Read Best Answer
Editorial / Best Answer
spartankiya- Member Since Feb-2007 | Feb 19th, 2007
MetaStage is a persistent metadata Directory that uniquely synchronizes metadata across multiple separate silos, eliminating re keying and the manual establishment of cross-tool relationships. Based on patented technology, it provides seamless cross-tool integration throughout the entire Business Intelligence and data integration life cycle and tool sets
spartankiya
- Feb 19th, 2007
MetaStage is a persistent metadata Directory that uniquely synchronizes metadata across multiple separate silos, eliminating re keying and the manual establishment of cross-tool relationships. Based o...
-
What does a Config File in parallel extender consist of?
Config file consists of the following. a) Number of Processes or Nodes. b) Actual Disk Storage Location.
Venkat Poonati
- Mar 29th, 2011
1. fastname-----usually networkname,It is physicall node name,That stages use to open ...
aladap12
- Jun 26th, 2009
Biggest strength of Datastage PX is its configuration file.. Datastage engine learns about size and shape of the system by reading configuration file.It contains Nodes and resource available for running the job.
-
ETL Tools Lookup Stage
By using the Lookup stage we will combine the data from multiple table. By using join stage also we will combine the data from multiple tables. Then what is the need of Lookup stage?
Venkat Poonati
- Mar 29th, 2011
There are three things to consider here1.memory usage As with joiner fewer rows has to be in memory at any time.This is not with lookup2.treating of rows with unmatched keys Joiner w...
narra satish
- Mar 21st, 2011
Join and lookup is differ in terms of memory usage. Join is light weight than lookup stage. Join stage won't use much system resources. While performing lookup will used more system resources. In case of lookup it fetch the whole data from refernce link into RAM memory and then it perform lookup.
DataStage Interview Questions
Read Best Answer
Ans