Why and When do we get the "Pipeline Broken Error" in Ab Initio

Alok S

• Jan 17th, 2020

When total number of process in a unix box connecting to a particular user reaches the limit, it throws the error "pipeline broken". Which means no more connection to the user can be made. So wait for sometime and re-login.

cutesree

 Profile Answers by cutesree Questions by cutesree

• Dec 29th, 2009

Pipeline broken error will actually indicates the?failure of a downstream component.It normally occurs when the database is?running out of memory?which makes?database components in the graph unavailable.

Answer Question  Select Best Answer

I have already created graph,and need to modify it so that output file gets page breaks. Any idea how can we implement page breaks in abinitio.

Anu

• Jan 10th, 2020

Sed G filename

swarna

• Apr 10th, 2014

I think they mean inserting new line character in unix or
meaning carriage return in windows environment.

Answer Question  Select Best Answer

What are the date functions in Abinitio & How to check a file is not corrupted in Abinitio?

Arun Ramaswamy

• Dec 3rd, 2019

There are lots of date functions in abinitio.
For checking if a file is corrupted try reading the file with dml and do a is_valid on the fields.

KV

• Aug 27th, 2016

Date functions- now(), datetime_add(),date_month()

Answer Question  Select Best Answer

Hi Experts,
I have to write the below floor function in reformat.
Floor(dp.lcl_Prod_cd/10)
The source column data type is string as shown {utf8 string("x01") lcl_prod_cd}, here column name is "lcl_Prod_cd ". now i have to apply the floor function on this like "Floor(dp.lcl_Prod_cd/10)". i was tried in different ways like i was changed target dml to decimal and tried as shown below....

Purva

• Oct 31st, 2019

Try using this:
out.lcl_prod_cd:: floor(decimal(10)in.lcl_prod_cd/10);

• Jul 4th, 2018

Hi Maheswari,
Can you please post the exact error. If it is type casting problem means you should convert input string value to decimal and then to integer, then it will work.

Let me know if am wrong.

Thanks,
Sivaramakrishna

Answer Question  Select Best Answer

What is the main difference between max core and max memory.
Where we use this commonly?

Rahul Rasal

• Jan 25th, 2019

Max core : component max-core parameter determines the maximum amount of memory the component will consume per partition before it spills to disk.
max memory : Enables the collection of memory usage data for components.

Answer Question  Select Best Answer

How can I set up email notification graph failure in Ab Iniio to send ,which should include error message and componet details which caused the failure

M Farhan

• Dec 31st, 2018

In end script for mpjret !=0
use mail command from unix

Kiran

• Apr 22nd, 2015

We can send the mail by using component of Send E-Mail notification.

Answer Question  Select Best Answer

What do you mean by Data Directory in abinitio?

Amol

• Dec 2nd, 2018

It is a directory used while creating the private projects, You would perform this by Command line or through GDE.

Answer Question  Select Best Answer

Raj

• Nov 20th, 2018

I am yet to work on Ab Initio. What I see from this thread is no one knows, the exact differences between the Roll Up and Aggregate apart from the Performance. Every transform will have its own capabi...

sateesh44

 Profile Answers by sateesh44 Questions by sateesh44

• Apr 12th, 2011

Aggregate and ROLLUP generates records that summarize groups of records. But Roll-up provides more control over record selection, grouping, and aggregation.

Answer Question  Select Best Answer

I have file containing 5 unique rows and I am passing them through SORT component using null key and and passing output of SORT to Dedup sort. What will happen, what will be the output.

Amaan Sajid Ansari

• Nov 14th, 2018

In Case of sort output would be same as input (unsorted) in case of dedup will depend on no of partition and the dedup method used. Please refer to the other similar dedup question i have explain it in that

san

• Aug 7th, 2018

1. In case of Null ket{} in sort component nothing will happen. 2. In case of Dedup sorted - keep - fist then you will get first record, Keep- Last then you will get last one, Keep- Unique only - in ...

Answer Question  Select Best Answer

If we give NULL key in scan component what will be the output and in dedup with keep parameter as unique?

Amaan Sajid Ansari

• Nov 14th, 2018

For unique only, the output for Dedup will be 0 records. While if you keep it to first or last it will be first or last record of each partition respectively.. While in case of scam it will treat the ...

Krushna

• Sep 6th, 2018

If key is {} in sort then Data will not get sort. It gives as it is data which is received from in port

Answer Question  Select Best Answer

I have a scenario like below:

Date Amt
01-01-2016 100
01-01-2016 300
02-01-2016 350
01-01-2016 200
02-01-2016 150
02-01-2016 400
01-01-2016 250
02-01-2016 250

I want to find top to transactions on each date:

Amaan Sajid Ansari

• Nov 14th, 2018

Use SORT---> SCAN (use scan select function and set the value as < =2)

Krushna

• Sep 7th, 2018

Sort on date and amount(descending) then use scan to assign number within group then filter records whose number

Answer Question  Select Best Answer

There is a SFTP component placed at last of the graph.
Lineage report keeps on failing for every run because, the destination server cannot be considered.
It was success for all other components/subgraps in that graph.

Could you please help me here how can I get lineage report success?

anthati

• Oct 25th, 2018

Use document component to set graph lineage

Answer Question  Select Best Answer

How do I add Delimiter to a file which has text in the below format:
(There are multiple rows in the file like below row)
246631balacl1Lavanya Balachandran20120120R7891Unknown-1Unknown

Munni

• Sep 17th, 2018

Read the file using fixed length DML and use delimited DML at the output

PJ

• Apr 17th, 2018

But you havent mentioned where you want to add delimiters in given data

Answer Question  Select Best Answer

How to generate DML from backend and front end?

inamdnik

 Profile Answers by inamdnik Questions by inamdnik

• Jul 3rd, 2018

Did you mean generate DML from UNIX(Back-end) and GDE(Front-end)?
If so then below is the answer
1. Generating DML from UNIX level
m_db gendml -table > $AI_DML/

2. Generating DML from Graph(GDE) level
You can create it from any components port tab--> Same As and Embedded radio buttons.

Answer Question  Select Best Answer

How can we fetch next record data for a particular field in ab initio?
i/p
id sal
101 500
102 800
103 900
105 200
o/p
id sal Pre_sal
101 500 0
102 800 500
103 900 800
105 200 900

Sai

• May 14th, 2018

use scan component we have to achieve this scenario and the parameter key is "NULL({})"and find the below explanation.Here i am taking the 2 temporary variables as a,b. type temporary_type = record ...

neha.nagare

 Profile Answers by neha.nagare Questions by neha.nagare

• Apr 26th, 2018

We can achieve this by using reformat component by maintaining current and previous value of the column sal and by updating variable prev_sal with previous record value of column sal. Please find...

Answer Question  Select Best Answer

What is the disadvantage of using multi-file if data volume is not much?

Jatin

• Apr 30th, 2018

1. When Graph starts there is time taken for start and shut down, so if data in multifile is not much use serial processing as over all time taken will not have any significant difference. 2. If mult...

Vinit

• Sep 26th, 2017

1. In multi files, there can be skew spillage due to imbalance in amount of CPU time
2. If data is not evenly partition, some of the partition can contain zero records and which will consume memory

Answer Question  Select Best Answer

jatin

• Apr 30th, 2018

1. In memory components where input not need not be sorted is selected like Rollup,join,scan,
2. Sort component

sonalika

• Sep 21st, 2017

Sort breaks pipeline parallelism

Answer Question  Select Best Answer

How to use rollup Component as Scan Component? On what condition Rollup component can be used as SCAN?

Jatin

• Apr 30th, 2018

Rollup will produce same number of columns as scan only in case when all the columns are selected as rollup key

Speedy

• Sep 22nd, 2017

As per my understanding, we cannot use rollup component as a scan component. The reason for this is Scan gets the cumulative data and can process the same number of output records as the number of inp...

Answer Question  Select Best Answer

I have a record like below:
avi,mahi,virat,mahi,avi,virat,mahi,virat,avi
Output should be:
avi1
mahi1
virat1
mahi2
avi2
virat3
mahi3
virat4
avi3
Please suggest how can I achieve this.

neha.nagare

 Profile Answers by neha.nagare Questions by neha.nagare

• Apr 26th, 2018

Flow I/p ->normalize ->scan->o/p We can achieve this by using normalize and scan component ,make vector by using string_split function and assign size of the vector to the length function of the no...

Sandeep

• Mar 25th, 2018

Use scan component using check sort attribute as false(maintain order-false)
then use name as key and for sequence number use count(in.name)

Answer Question  Select Best Answer

I have 100 records in input file if I run graph first time first 20 records will go to output, in next run 21 to 40 records will go to output, like this in 5th run 81 to 100 records will to output. Please let me know how can I achieve this?

Thanks in advance

siva

• Apr 17th, 2018

Use the Partition by round robin component and set the block size 20

Natrayan

• May 16th, 2017

We can use the Partition by range component to segregate the record based on record number

Answer Question  Select Best Answer