How to flush data stored in SSIS buffer

Rafal Chojnacki


Hi,

I am working on my own data flow source component. Here is a fragment of this component code:

public override void PrimeOutput(int outputs, int[] outputIDs, PipelineBuffer[] buffers)
{
PipelineBuffer selectedBuffer = buffers[0];
string message;

while ((message = GetMessage()) != null)
{
selectedBuffer.AddRow();
selectedBuffer.SetString(0, message);

// how to flush data here
}

selectedBuffer.SetEndOfRowset();
}

private string GetMessage()
{
// we are retrieving some message here, this is a long-term process
}

When a new row is added by this component to the buffer then this row is not immediately available to the next component in data flow. It is possible to configure SSIS in that way that each row is immediately sent to the next data flow component If no also please inform me about that.

Thanks,
Rafal

JayH


Rows are not handled individually in the pipeline, but are grouped onto buffers. Your rows will not be available to downstream components until the buffer is full or there are no more rows upstream. By default buffers hold 10MB or 10,000 rows. You might be able to cause rows to be available immediately by setting the DefaultBufferMaxRows to 1, but this is not recommended and will probably have a huge negative impact on performance.

Rafal Chojnacki

When DefaultBufferMaxRows is set to 1 problem still exists.

I used SSIS for batch-oriented data integration always. Unfortunately, now I need a ETL-like platform for real-time data integration. It seems that SSIS is not good solution for real-time data integration even if it is extended by specialized data source adapter. Do you agree

JayH

Real-time is a difficult term to define. I think SSIS can push data to a destination fast enough to qualify as real-time for most applications. I think what really determines the appropriateness of SSIS for a real-time application is what your source data is. Scenarios that I've been involved with have had message-based sources, and in those cases I've skipped SSIS and consumed the source data with Service Broker.

rafal.chojnacki

Thank you for your help