"Paralellizing" file loads

I have a package that reads a table that has a list of files that I need to load into a table that arrive every night. These files range from 50mb to 1.5gb, The entire process to load and transform is taking about 50mins, which is about a 300% increase from our current production environment. However I am looking at ways to improve performance by loading all of the data into the staging table at the same time.

Is this possible, and do you guys think it would improve performance significantly?

Can you provide more details about the source target picture?

So, many files going into a single staging table?

If so, You may want to try having several threads reading from the files and inserting into the target table; perhaps using multiple dataflows.

|||

The source are about 27 files arriving from a customer to a network drive. Currently we start the package when all the files have arrived and we load 1 file at a time into a table using a simple FileConnection->OLE DB Destination.

I have a Foreach Container (files) that loops on an ADO.NET Recordset. Inside the Forloop container it starts a package which loads the file.

So my question is how could I launch that package from within my package "asynchronously"?

|||Unforntunatly, Using a For Each loop will not help you on processecing several files at the time, which I think is the only way to speed the process up.|||

There is a possibility here, Rafael and Jwelch have given a bunch of clues in previous forums to speed up, it all depends and I have clues too,

1) Incase you have a file meant for a single table you can have few parallel dataflow extracts.

2) If there are list of files populated into each table then we can group the file according to the pattern (use wild cards or expression to assign file format) and then run few Parallel extracts using ForEach loop for each of the file groups

3) If you are loading only a single table then parallel insertion wont help, because table lock might apply when rows are getting imported from a file.

Thanks

Subhash Subramanyam

"Index was out of range" when two cells are merged

I receive this error during rendering when I have two cells merged together:

Error Snippet

Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index

When I "Split Cells" on the offending cells, it starts to work. This report renders without error when run on the RS server. This error only occurs when running the report locally on the "Preview" tab in the report designer.

I have closed the IDE and deleted the *.data files and restarted with the same results. Is there anyway to get more information about the error to help debug the problem?

Thanks!

~Jon

I would try copying the report content from your current report to the clipboard, deleting all of the content, and then pasting it back. This should allow the text boxes and such to be rebuilt and may clear up the issue. Keep a copy of the original RDL and compare the new (hopefully working) to the old.

Larry

|||

I found something...

It appears that I cannot merge cells in the top row in the Table Header. I can have merged cells in rows below the top row. Anytime I merge the cells on the top header row I get this index out of range error. I have no idea why. I tried it with another report and it works fine. It is just this particular report. The report is pretty simple. It does call a couple of subreports that are in the header. The crazy thing is that it renders with no problem on the RS server.

For now I just added a row at the top that doesn't contain any merged cells so that I can Preview locally as I work on the report.

~Jon

<SelectParameters> & checked items from CheckBoxList

Column1 in table 1 has a range from 1 to 5 (int)

A CheckboxList displays these 5 items. When checked (for example 1 and 4) I want a gridview with sqldatasource=sqlProducts to display the records where column1 values 1 or 4.

When I use the code below I only get the records where column1 values 1...

<asp:SQLDataSource id="sqlProducts" Runat="Server" SelectCommand="Select * From Table1 where Column1 IN (@.CBLchecked_items)" ConnectionString="<%$ ConnectionStrings:pubs %>">
<SelectParameters>
<asp:ControlParameter Name="CBLchecked_items" ControlID="CBL" propertyname="SelectedValue" type="String" />
</SelectParameters>
</asp:SQLDataSource>

Hi~

SelectedValue returns the first selected value .

You can iterate to get all selected values:

For i = 0To Check1.Items.Count - 1If Check1.Items(i).SelectedThen' List the selected items s = s & Check1.Items(i).Text s = s &","End If Next

Hope it helps.|||

I got the problem to get the checked item in CheckBoxList here is the code :

foreach (ListItem itemin cblRoles.Items){if (item.Selected)// here the item.Selected always = false {// do something } }