rockeye

Hi,

I am a newb in c# and I am trying to convert an array pointed by an IntPtr ip to an Int32[] data WITHOUT copy.

I have tried Marshal.PtrToStructure(ip, data), but it raises an Argument exception because data seems to be unblittable but this http://msdn2.microsoft.com/en-us/library/75dwhxf7(VS.80).aspx tell the contrary.
I found a lot of threads for converting Array in IntPtr, using a fixed block, but I never found a code sample showing the inverted process...

Can someone help me Thanks!







Re: Visual C# General Converting from IntPrt to int32[]

TaylorMichaelL

The documentation is not contradictory. Arrays of PODs are blittable. PtrToStructure won't work because a managed array is not the same as an array of values. In all cases though it is important that you know how much memory the IntPtr points to in advance.

You are under the impression that managed arrays work just like unmanaged C arrays where a pointer and an array are the same but this is not necessarily true. To make this assumption is just asking for trouble. An IntPtr is completely different than a managed array. For one IntPtr is a value type and therefore it is allocated on the stack. Arrays are always reference types (although they can hold value types) so they are allocated on the heap. Starting to see the problem yet

You can't convert one to the other without copying short of a typecast (which might not even work). You have no choice but to use Marshal.Copy to copy the unmanaged data back to a managed array.

Michael Taylor - 6/8/07

http://p3net.mvps.org





Re: Visual C# General Converting from IntPrt to int32[]

IsshouFuuraibou

Alternatively, how are you populating the IntPtr with the pointer to the array If you're P/Invoking to get it, you may want to experiment more with the P/Invoke and do some specific marshaling an let the marshaler handle the conversion. There are MarshalAs types for dealing with unmanaged arrays being returned.





Re: Visual C# General Converting from IntPrt to int32[]

el_d

I try to do exactly the opposite. Access to a managed byte[] from unmanaged c++.

And it would be a LOT better no to have to copy the data.

You seem to say that this is not possible to work without copy. But dotNet exposing COM does!

I wrote a managed component exposing a managed byte[] to COM, with the following methods.

byte[] GetData() (in C#)

void CheckDataChanges()

From a native c++ application, I acces the managed object through COM.

GetData() returns a SAFEARRAY. I modify the data using classic native arrays access, and then I call CheckDataChanges, and the managed code sees the modified data.

So there was no copy. And also no memory alignement problems on more than 400MB!

Any idea how to do the same magic (managed and unmanaged arrays referring the SAME data) without COM





Re: Visual C# General Converting from IntPrt to int32[]

TaylorMichaelL

You are confusing what the OP wanted to do with what you are trying to do. If you use P/Invoke to marshal a byte array to unmanaged code the marshal handles the copying of the data from the managed array to the unmanaged array. Refer to the article Copying and Pinning in MSDN for an example of how this works with arrays.

COM is quite a bit different. In COM you can only use C++-style arrays when working with inproc COM components. In all other cases, and when working with VB and other IDisposable-based languages, you must use a SAFEARRAY. A SAFEARRAY is not a normal array either. SAFEARRAYs are allocated in system-shared memory such that they can be accessed across processes. They must be allocated and deallocated manually. If you use SAFEARRAYs then you don't need to copy the array because it is in shared memory. However SAFEARRAYs must be accessed using the SafeArrayGetElement function rather than treating them like a normal array (which they aren't). In reality they are probably contigous in memory to keep them as small as possible but padding for alignment and whatnot can easily mess this system up. You can use the SafeArrayAccessData method to get access to the underlying pointer for basic array work but this pointer resides in system memory and has nothing to do with managed code.

As for your code yes when you define a COM interface in .NET it will generate the interface using SAFEARRAY because, by default, the interface must be IDispatch-compliant. This mandates a SAFEARRAY because C-style arrays aren't supported. Using SafeArrayAccessData you can access the underlying pointer to the array and work with it in native code.

You can continue to access a SAFEARRAY in .NET but you will need to use the Win32 APIs to do so. Otherwise you'll get an IntPtr which points to system memory. This system memory is not a managed array. If you want to treat it as a managed array then you have to copy it into a managed array.

Michael Taylor - 6/11/07

http://p3net.mvps.org





Re: Visual C# General Converting from IntPrt to int32[]

el_d

A litte bit confusing for me... Wink

TaylorMichaelL wrote:
If you use SAFEARRAYs then you don't need to copy the array because it is in shared memory. However SAFEARRAYs must be accessed using the SafeArrayGetElement function rather than treating them like a normal array (which they aren't). In reality they are probably contigous in memory to keep them as small as possible but padding for alignment and whatnot can easily mess this system up. You can use the SafeArrayAccessData method to get access to the underlying pointer for basic array work but this pointer resides in system memory and has nothing to do with managed code.

Does this mean that SafeArrayGetElement does a copy of the safe array data so that it can be accessed as a normal array, and that SafeArrayUnaccessData does copy the changes back

TaylorMichaelL wrote:
You can continue to access a SAFEARRAY in .NET but you will need to use the Win32 APIs to do so.

You mean using SafeArrayGetElement from managed code with PInvoke

TaylorMichaelL wrote:
Otherwise you'll get an IntPtr which points to system memory. This system memory is not a managed array. If you want to treat it as a managed array then you have to copy it into a managed array.

And this would be pretty slow to create a managed class deriving from IList<Byte> and using this IntPtr or SafeArrayGetElement, right I just tried this (a managed IList<Byte> derived class that uses a native array using IJW), and array access is 10 times slower that a managed array!

So the method of copying would be faster than accessing a native array with IJW





Re: Visual C# General Converting from IntPrt to int32[]

el_d

And what of this code (posted by Nicholas Paldino [.NET/C# MVP] on http://www.dotnet247.com/247reference/msgs/58/292873.aspx):

[DllImport("myfunc.dll")]
private static extern unsafe int dummy(byte *array);

Then, in your code, you would call in an unsafe block, like this:

// The byte array.
byte[] bytes = new byte[5000];

// Declare an unsafe block, or you can put this on the method.
unsafe
{
// Fix the byte array.
fixed (byte *array = bytes)
{
// Make the call here, passing in the array.
int retVal = dummy(array);
}
}

I just tested it with an unmanaged C++ fonction attained through IJW instead of PInvoke, and it works perfect. My managed app can make a call to a native fonction that accesses and modifes the managed array. And without copy (performance is this time better than managed access to the managed array).

According to ms-help (fixed Statement (C# Reference)), this "fixed (byte *array = bytes)" is official and is not a hack.

It is just not said that the byte* may be used for native calls (but I do not see what else it could be used for). Can I assume this, or do I have to fear for memory alignment problems on some systems

Thanks!





Re: Visual C# General Converting from IntPrt to int32[]

el_d

Ok, the equivalent is documented in ms-help (pin_ptr) for managed C++.

A managed array is modified by a native function without any copy. That saves me! Smile

// pin_ptr_1.cpp
// compile with: /clr 
using namespace System;
#define SIZE 10

#pragma unmanaged
// native function that initializes an array
void native_function(int* p) {
  for(int i = 0 ; i < 10 ; i++)
  pIdea = i;
}
#pragma managed

public ref class A {
private:
  array<int>^ arr;  // CLR integer array

public:
  A() {
   arr = gcnew array<int>(SIZE);
  }

  void load() {
  pin_ptr<int> p = &arr[0];  // pin pointer to first element in arr
  int* np = p;  // pointer to the first element in arr
  native_function(np);  // pass pointer to native function
  }

  int sum() {
   int total = 0;
   for (int i = 0 ; i < SIZE ; i++)
     total += arrIdea;
   return total;
  }
};

int main() {
  A^ a = gcnew A;
  a->load();  // initialize managed array using the native function
  Console::WriteLine(a->sum());
}




Re: Visual C# General Converting from IntPrt to int32[]

TaylorMichaelL

I will summarize my recommendations.

  1. Do not use SAFEARRAY unless you have to do interop with COM. It simply isn't worth the overhead.
  2. Use Marshal to copy data from managed to unmanaged and vice versa when using P/Invoke and the default marshaling does not give you what you want.
  3. If you are going to be doing a lot of interop work and you know C++ then use C++/CLI to allow you to seamlessly use managed and unmanaged code together. The compiler will generate the code on the back end to do the heavy lifting.
  4. Be careful about pinning pointers in memory. This can cause memory fragmentation. Pin only when necessary.

The overhead for copying blocks of memory from one area to another are almost non-existent except when dealing with really large blocks. Modern CPUs can copy entire pages of memory relatively quickly and memory is cheap. Unless you're talking about moving thousands of elements in an array copying is not going to cause you any performance issues but pinning or SAFEARRAYs might. I recommend that you implement your algorithms using the most efficient and simple algorithm you can. Then use a profiler to identify the bottlenecks. If memory copies are truly causing you performance problems (which I doubt) then optimize the code. The general rule is right the code correctly and then optimize the 10% of the code that is causing performance problems.

Michael Taylor - 6/12/07

http://p3net.mvps.org





Re: Visual C# General Converting from IntPrt to int32[]

el_d

Hello. Thanks for your answer.

I have to do interop with several arrays of more than 12MB each. And this could increase soon, as digital cameras are always progressing...

That's why I fear copying my data (and in fact copying twice : the native part must modify the data coming from the managed app, and the managed app must then get these modifications).

You say "Be careful about pinning pointers in memory. This can cause memory fragmentation.". I'm not getting it... Could you be more precise What are the drawbacks of this method





Re: Visual C# General Converting from IntPrt to int32[]

TaylorMichaelL

Personally I'd evaluate whether you are really gaining anything by using C# or .NET. C++ would be the fastest, easiest route. If you really want to use .NET then I'd recommend the C++/CLI route as you can do unmanaged work quickly (without copying) while still having access to .NET (with copying and thunking) if you really need it.

What I meant about pinning is that when you pin a managed object in memory the GC can no longer move it around. When a GC occurs the GC will remove unreferenced objects. It will then move the remaining objects in memory up to make room for more data. This avoids memory fragmentation issues that can occur when memory is allocated and released over time. You've probably seen it before. You might have 100MB of available memory but you can't allocate a 1MB buffer because so many little memory allocations have occurred over time that there is not 1MB of contiguous memory available to allocate. The GC avoids this.

With pinning the GC can't move the memory block so it causes fragmentation. If you do this to a lot of blocks then fragmentation starts to become an issue. However if you are dealing with 12MB blocks then you will be using the LOH (large object heap). This particular heap is designed for large objects and doesn't move memory around during GC.

So, in summary, I'd recommend using native C++ unless you need managed support in which case I'd recommend using C++/CLI to maximize memory and speed. Use .NET/managed code only when you absolutely need it and avoid thunking as much as possible.

Michael Taylor - 6/12/07

http://p3net.mvps.org





Re: Visual C# General Converting from IntPrt to int32[]

el_d

Ok, thanks for your (very complete) answer.

We want to use dotNet for productivity reasons, as we have a lot of interface to deal with.

Memory fragmentation should not be an issue as we can preallocate the (few) buffers and make sure they do not stay pinned too long.

For documentation I include the following link that goes in the detail of GC heap fragmentation :

https://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx





Re: Visual C# General Converting from IntPrt to int32[]

HWM

Hi

I have read through this thread, and you are dealing with an issue that needs caution!

However, I did notice that you seem to want to access/manipulate an unmanaged array of Int32 directly in managed code

If this is correct, then may I suggest you visit our website and grab a copy of Beta 1 of our product

So far as your issues goes, we directly support the ability to create what appear to managed arrays of Value types.

However the actual data is stored in shared memory and managed by the product's memory management kernel.

We acheive this by using a .NET class that masquerades as an array, yet internally access the shared memory directly.

e.g.

Shared<Int32> fast_array = new Shared<Int32>(shared_folder,

100,

"fast_array",

256,

512);

I can't dwell on this too much here, but after this has excuted, "fast_array" may be manipulated intuitively as follows:

if (fast_array[12, 45] == 12)

....

or

fast_array[0, 0] = 120;

or

foreach (Int32 val in fast_array)

...

The data is accessed directly by the implementation of the "Shared<T>" class. You may create arrays with any number of dimensions (the example has two dimensions, one being 256 the other 512).

Of course the other feature of all this is that the data is truly shared, any other thread or process may create the same named array and they will all be accessing the very same data in real-time, furthermore if your code crashes, no-matter the data is all autimatically preserved, in fact you never need to persist the data in your array, it is by its very nature persistent, all you need ever do is just declare it and access it!

Do have a look and do contact us for more info, Beta 2 is due soon and has many more poweful capabilities.

In fact I see no reason why you cant do a similar thing and create a class that looks like an array, but stores the Int32 * pointer inside itself. You could then do a similar thing to what we do and let the .NET code look like its accessing an array, when its just calling (ultimately) an indexer method that does some pointer arithmetic and accesses the stored data directly.

Rgds