skrewed


Hi,

Firtsly - I am new to SSIS if my approach could be improved then I welcome suggestions.

Scenario: I have a large SSIS package that consolidates / summarizes work week information from several data sources. Currently each data flow task in the control flow calculates the from and to date that is filtered on, for example:

DECLARE @FromDT AS DATETIME
SET @FromDT = CAST(FLOOR( CAST( DATEADD(D, -7, GETDATE()) AS FLOAT ) ) AS DATETIME)

DECLARE @ToDT AS DATETIME
SET @ToDT = CAST(FLOOR( CAST( GETDATE() AS FLOAT ) ) AS DATETIME)

I would like to remove these statements that appear in most steps and replace them with a global variable that is used throughout the package. This statement would only appear once & it would make the package much easier to run after failure etc.

Problem: I am using Data Reader Source with the 'SQLCommand' property specified. It looks like parameters are only supported if an OleDB connection is used

So I switched to an OleDB connection and no parameters are recognised in the string - a forum search reveals that parameters in sub queries are not always found properly. The solution to this problem appears to be, to set 'Bypass Prepare' to True but this is a property for the Execute SQL task, not the Data Flow Task source.

Questions:

  1. Does the Data Reader Source control from Data Flow Source toolbox section support parameters
  2. Can anyone suggest a fix to the OleDB Source issue with Parameters
  3. Is there a better way to solve my problem e.g. Using Execute SQL Task instead of Data Flow tasks etc

Example SQL:

This SQL is an example of the SQL for the OleDB Data Source (within a Data Flow task)

------------------------------
--RADIUS LOGINS
------------------------------
DECLARE @FromDT AS DATETIME
SET @FromDT = CAST(FLOOR( CAST( DATEADD(D, -7, GETDATE()) AS FLOAT ) ) AS DATETIME)

DECLARE @ToDT AS DATETIME
SET @ToDT = CAST(FLOOR( CAST( GETDATE() AS FLOAT ) ) AS DATETIME)

DECLARE @Attempts AS BIGINT
SET @Attempts =
(SELECT COUNT(*)
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN @FromDT AND @ToDT)

DECLARE @Failures AS BIGINT
SET @Failures =
(SELECT COUNT(*)
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN @FromDT AND @ToDT
AND Authen_Failure_Code IS NOT NULL)

DECLARE @Successes AS BIGINT
SET @Successes = @Attempts - @Failures

DECLARE @OcaV1Hits AS BIGINT
SET @OcaV1Hits = (SELECT COUNT(DISTINCT LoginName)
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN
@FromDT AND @ToDT
AND EAPTypeID = 25)

DECLARE @OcaV2Hits AS BIGINT
SET @OcaV2Hits = (SELECT COUNT(DISTINCT LoginName) AS OcaV2Hits
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN
@FromDT AND @ToDT
AND EAPTypeID = 13)

SELECT
@Attempts AS ConnectionAttempts,
@Failures AS ConnectionFailures,
(CAST(@Successes AS DECIMAL(38,2)) / CAST(@Attempts AS FLOAT) * 100) AS SuccessRate,
@OcaV1Hits AS OcaV1Hits,
@OcaV2Hits AS OcaV2Hits

Please remember, I'm new to SSIS - so be detailed in your response. Thanks for your help!




Re: OleDB Source and Parameters

Rafael Salas


skrewed wrote:

Hi,

Firtsly - I am new to SSIS if my approach could be improved then I welcome suggestions.

Scenario: I have a large SSIS package that consolidates / summarizes work week information from several data sources. Currently each data flow task in the control flow calculates the from and to date that is filtered on, for example:

DECLARE @FromDT AS DATETIME
SET @FromDT = CAST(FLOOR( CAST( DATEADD(D, -7, GETDATE()) AS FLOAT ) ) AS DATETIME)

DECLARE @ToDT AS DATETIME
SET @ToDT = CAST(FLOOR( CAST( GETDATE() AS FLOAT ) ) AS DATETIME)

I would like to remove these statements that appear in most steps and replace them with a global variable that is used throughout the package. This statement would only appear once & it would make the package much easier to run after failure etc.

Problem: I am using Data Reader Source with the 'SQLCommand' property specified. It looks like parameters are only supported if an OleDB connection is used

So I switched to an OleDB connection and no parameters are recognised in the string - a forum search reveals that parameters in sub queries are not always found properly. The solution to this problem appears to be, to set 'Bypass Prepare' to True but this is a property for the Execute SQL task, not the Data Flow Task source.

Questions:

  1. Does the Data Reader Source control from Data Flow Source toolbox section support parameters
  2. Can anyone suggest a fix to the OleDB Source issue with Parameters
  3. Is there a better way to solve my problem e.g. Using Execute SQL Task instead of Data Flow tasks etc

Example SQL:

This SQL is an example of the SQL for the OleDB Data Source (within a Data Flow task)

------------------------------
--RADIUS LOGINS
------------------------------
DECLARE @FromDT AS DATETIME
SET @FromDT = CAST(FLOOR( CAST( DATEADD(D, -7, GETDATE()) AS FLOAT ) ) AS DATETIME)

DECLARE @ToDT AS DATETIME
SET @ToDT = CAST(FLOOR( CAST( GETDATE() AS FLOAT ) ) AS DATETIME)

DECLARE @Attempts AS BIGINT
SET @Attempts =
(SELECT COUNT(*)
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN @FromDT AND @ToDT)

DECLARE @Failures AS BIGINT
SET @Failures =
(SELECT COUNT(*)
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN @FromDT AND @ToDT
AND Authen_Failure_Code IS NOT NULL)

DECLARE @Successes AS BIGINT
SET @Successes = @Attempts - @Failures

DECLARE @OcaV1Hits AS BIGINT
SET @OcaV1Hits = (SELECT COUNT(DISTINCT LoginName)
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN
@FromDT AND @ToDT
AND EAPTypeID = 25)

DECLARE @OcaV2Hits AS BIGINT
SET @OcaV2Hits = (SELECT COUNT(DISTINCT LoginName) AS OcaV2Hits
FROM dbo.Radius_Login_Records
WHERE LoggedAt BETWEEN
@FromDT AND @ToDT
AND EAPTypeID = 13)

SELECT
@Attempts AS ConnectionAttempts,
@Failures AS ConnectionFailures,
(CAST(@Successes AS DECIMAL(38,2)) / CAST(@Attempts AS FLOAT) * 100) AS SuccessRate,
@OcaV1Hits AS OcaV1Hits,
@OcaV2Hits AS OcaV2Hits

Please remember, I'm new to SSIS - so be detailed in your response. Thanks for your help!

In general you can place the query in a varaibale and then use that variable in your datareader (thrugh an expression http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=1283829&SiteID=1) or OLEDB command to provide the SQl query.

If you need to run the same query several times becasue you have more than one (identical) source; then you may try to use a for each loop container and then place the data flow logic inside. Please provie more details about the nature of your process.







Re: OleDB Source and Parameters

skrewed

Hi Rafael,

Sorry for the delay in my response, I was travelling. Thanks for your answer - this has worked.

I now have a script task at the start of my package that defines the global to/from dates - individual data flow tasks within the package use these values. If execution now fails and I need to run the package on monday or tuesday for past dates, thenI can quickly edit the variable.

The expression builder is pretty powerful, although I have some thoughts on this approach:

  • Be wary of Data Types - for example, variable data type is a string, haven't found a good way to cast the data type in the expression without causing errors (cast operator documentation not very good)
  • This is pretty cryptic when you think about it - placing a variable into a query required me to build an expression...seems complicated for what I'm tyring to achieve
  • Also, "Evaluate expression" button/function - glad this is there, a must have piece of functionality

Thanks for your help!!






Re: OleDB Source and Parameters

Rafael Salas

skrewed wrote:

Hi Rafael,

Sorry for the delay in my response, I was travelling. Thanks for your answer - this has worked.

Great!

skrewed wrote:

  • Be wary of Data Types - for example, variable data type is a string, haven't found a good way to cast the data type in the expression without causing errors (cast operator documentation not very good)

And you won't find a way to cat it. The expression editor expects a string ALWAYS. You just need to make sure that you get the expect result when you evaluate the expression.

skrewed wrote:

  • This is pretty cryptic when you think about it - placing a variable into a query required me to build an expression...seems complicated for what I'm tyring to achieve

It can look complicated but it is a very efective and powerfull feauture to set dynamic values to objetcs' properties.

skrewed wrote:

  • Also, "Evaluate expression" button/function - glad this is there, a must have piece of functionality

I agree, it is verry helpful. There are some request from the users to include and upfront visula indicator when a property is being affected by an expression. Right now, the only way know if there is an expression is digging into the expression property.






Re: OleDB Source and Parameters

jwelch

Rafael Salas wrote:

I agree, it is verry helpful. There are some request from the users to include and upfront visula indicator when a property is being affected by an expression. Right now, the only way know if there is an expression is digging into the expression property.

That's one of the features soon to be included in BIDS Helper (http://www.codeplex.com/bidshelper).