Loading multiple flat files with parameter file

Reply from Jatin | posted Oct 24, 2007 | Replies (2)

0
Please Sign-in or Register to vote
Working
Use Source file type as Indirect. Give filename as your workflow name i.e. workf
low1.dat
Write a shell script that will be called from pre-session command.
Script:
rm <source file dir>\workflow1.dat
ls > <Temp dir>\workflow1.dat
(If you use Source file dir instead of Temp dir, workflow1.dat will also be incl
uded in your file list and session will fail at that point)
mv <Temp dir>\workflow1.dat <source file dir>\workflow1.dat
********************************************************************************
*******************************************************************

Merge multiple Flat files

Reply from Nelrick Rodrigues | posted Oct 17, 2010 | Replies (5)

0
Please Sign-in or Register to vote
Working
Hi Jagan,
By merging multiple flat files, I assume you mean loading the data from all the
100 flat files into either a flat file or a relational table target.
The solution for this is to use a file list. But for this to work all the Flat F
iles should be having the same definition i.e. the data types, no. of columns, a
nd the type of flat file (delimited or fixed width) should be the same.
For this to be implemented, you need to import only one file definition in the S
ource Analyzer.
Create the desired mapping with this source.
Next, create a .txt file (file list) that will contain the path of all the 100 f
iles on separate lines. For instance, if you are using the ../infa_shared/SrcFil
es folder to place your flat files F1, F2,...F100, you can create the file list
file in this same directory and the entries in it would be as:
F1
F2
.
.
F100
Next, in order for this to work, we need to change the session properties.
Go to Edit Tasks > Mapping tab.
Click on Sources > Flat File definition in the left pane.
In the right pane, under properties, change the following:
Source filetype ---> Indirect
Source file directory ---> Directory of the file list (if you place it in the Sr
cFiles folder, the default entry i.e. $PMSourceFileDir\ need not be changed)
Source filename ----> Name of the file list file (.txt file).
In case, you have any problems let me know.
Regards,
Nelrick
********************************************************************************
**************************************************************************

If no duplicates in flat file then tgt flat file should be entered with 'No Dupl
icates in file'
Reply from Rajiv R Chitlur | posted Oct 8, 2008 | Replies (4)

0
Please Sign-in or Register to vote
Working
Hi,
Follow the following method :-
1. Add Source Qualifier Tfn
2. Add Expression Tfn
3. Add Sequence generator and drag port seq.nextval to the expression. Set Reset
option in the sequence generator
4. Add sorter transformation and connect ports from expression and check option
distinct in sorter properties
5. Add first aggregator1 transformation and connect ports from sorter. Create on
e output variable which computes count(*) value. This will give you number of re
cords coming from source
6. Add second aggregator2 transformation and connect ports from expression. Crea
te one output variable which computes max(seq) value. This will give you total n
umber of records coming from source.
7. Connect output ports from both aggregators to union transformation.
8. Add expression tfn and compare previous value with current value. If both the
values are same then create one output port in the expression transformation an
d pass message 'there are no duplicates' to the target.
Thanks,
Rajiv

Try this.
Expression :
ID (I/O)
VAR_ID (V) = ID
VAR_COUNT (V) = IIF(ISNULL(PREV_ID) OR PREV_ID <> VAR_ID, 1,2)
OUT_COUNT (O) = VAR_COUNT
OUT_VAL (O) = IIF(VAR_COUNT=1, NULL, 'No Duplicates in the flat file')
PREV_ID (V) = VAR_ID
Next is Sorter:
OUT_COUNTER (I/O) -- Sort on Ascending
OUT_VAL (I/O)
Next is Aggregator:
OUT_COUNTER (I/O)
OUT_VAL (I/O)
(Do not select Group By check box. It will return last record only)
Next is Filter:
OUT_COUNTER (I)
OUT_VAL (I/O)
Filter condition : OUT_COUNTER=2
Take output from Filter to Target
~Jatin