Wibs2

Hi,

The following code takes a csv file from disk (65,536 values, no spaces, no text delimters, all on one line), and converts the values into a byte array.

Private Sub csv2Byte(ByVal csvFilename As String)
Using csvReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(csvFilename)
csvReader.TextFieldType = FileIO.FieldType.Delimited
csvReader.SetDelimiters(",")
Dim csvFile As String()
Dim n As Integer
Dim csvValue As String
Dim myBytes() As Byte
Dim hexValues() As String
Dim index As Integer = -1
While Not csvReader.EndOfData
Try
csvFile = csvReader.ReadFields()
For Each csvValue In csvFile
index += 1
n = Convert.ToInt32(csvValue)
'Convert 'n' to a BYTE and add it into the array myBytes()
'at index position pointed to by variable 'index'
myBytes(index) = CByte(n)
hexValues(index) = Hex(n)
MsgBox("As text = " & csvValue & "; and as an integer = " & n.ToString)
Next
Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
MsgBox("Line " & ex.Message & "is not valid and will be skipped.")
End Try
End While
End Using
End Sub
End Class

However, during debugging and after a pause the following error message pops up:

The CLR has been unable to transition from COM context 0x1a1418 to COM context 0x1a1588 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment (STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.

Can anyone decypher this error message, and more importantly, suggest what may be causing it. The code appears reasonable.

Thanks

Wibs


Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Wibs2

I should have added that the app does not get as far as any MsgBox.




Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Wibs2

More info:

I generated a much smaller csv file of 255 values. the code processed it almost instantaneously. I then generated a csv file of 10K values, it took about 6 seconds to process. So, 64K values should be about 40-50 seconds, right Nope, it crashes before it gets there, after about 1 minute.




Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Tall Dude

You get that message during long running operations

while debugging. It is gently suggesting that even in real

time, this will be a slow operation and you might want to consider

using threading.

Now; If you have no delimiters, why are you trying to use code

that handles delimited text

Why not just read the whole file into a byte array and work

with it from there

Dim look() As Byte = System.IO.File.ReadAllBytes("c:\look.bin")

Also, for our information, what does a few bytes of this

file look like and what are you trying to translate these

bytes into






Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

spotty

Does this only occur in the debugger or does this occur when running the executable.



Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Tall Dude

If you do have single character text, comma delimited

(all in one line.) The code below does not give your error

on my test.

Public Class Form1

Dim look3() As Byte

Dim hexValues() As String

Private Sub Button1_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles Button1.Click

Dim sb As New System.Text.StringBuilder

' Create a dummy CSV file

For z As Integer = 1 To 65536

sb.Append("Z,")

Next

Dim testTxt As String = sb.ToString

' Remove the last comma

testTxt = testTxt.Remove(testTxt.Length - 1)

System.IO.File.WriteAllText("c:\look.csv", testTxt)

Dim n As Integer

'read in the dummy file

Dim look As String = _

System.IO.File.ReadAllText("c:\look.csv")

Dim look2() As String = Split(look, ",")

ReDim look3(look2.Length - 1)

ReDim hexValues(look2.Length - 1)

For x As Integer = 0 To look3.Length - 1

n = Convert.ToInt32(CChar(look2(x)))

' update the byte array

look3(x) = CByte(Microsoft.VisualBasic.AscW(look2(x)))

' update the hex array

hexValues(x) = Hex(look3(x))

'MsgBox("As text = " & look2(x) & "; and as an integer = " & n.ToString)

Next

End Sub

End Class






Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

spotty

You hit the nail on the head as to what my question was eluding to. This message pump error is to do with the debugger and would probably not occur in the runtime.

I'm curious also why your byte array is comma separated. They may be a reason but it may be because in your previous posts with byte arrays you are seeing the code to populate the byte array as comma separated

ie. Dim x as byte() = {1,2,3}

If you reading and writing byte arrays to files - use the My.computer.system.ReadAllBytes and My.Computer.WriteAllBytes to read and write binary data such as byte arrays. These are much quicker and efficient than reading in text and parsing the string values to a byte array - (very slow process for bit arrays)





Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Wibs2

Hi spotty, TallDude,

I tried spotty's suggestion, and Built the apllication, and indeed it does not crash. Strange thing though, although it takes only 6 seconds to process a file with 10K values, it took about 3 minutes to process 64K values.

TallDude, when I said it was a csv, it is, and when I said 'no text delimiters', what I meant was that the format of the file (opened in Notepad, for example) is:

6,7,7,7,8,8,8,9,10,12,12,13 etc

Not: "6","7","7","8" etc (as some csv formats are).

The code is standard code as given in both msdn and in the VB 2005 textbooks (and although there several warnings about limitations and errors, there is no mention made of limitations on length of file). I just can't believe that my 3GHz, Pentium4, 2Gb memory machine is struggling for 3 minutes with a 229K text file.

Is there a faster way






Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Wibs2

Hi TallDude,

Your code generated letters, whereas mine generated numbers. Try this and see if you get the same. A Form with a Button1 and a ProgressBar1 on it: this is my code for generating the test data file (random numbers between 0 and 255), comma-separated. Then try the test file that this generates.

Public Class Form1

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

Dim RandomClass As New Random()
Dim RandomNumber As Integer
Dim rn As Byte
Dim rnArray(65535) As Byte
Dim ReadDelimitedString As String = ""

For x As Integer = 0 To 65535
RandomNumber = RandomClass.Next(0, 255)
rn = CByte(RandomNumber)
rnArray(x) = rn
ReadDelimitedString += rnArray(x).ToString
ProgressBar1.Value = Fix(x / 655.35)
If x <> 65535 Then
ReadDelimitedString += ","
End If
Next
ProgressBar1.Value = 0
My.Computer.FileSystem.WriteAllText("c:\64kRandom.csv", ReadDelimitedString, False, System.Text.Encoding.ASCII)
MsgBox("CSV File successfully generated.")
End Sub
End Class

Wibs




Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

jo0ls

If all you are trying to do is to save the array to disk then use a binaryFormatter to serialize it.

At the moment you are converting (extra clock cycles to convert) each byte to Text, which can use 4 bytes to store 1 byte value in the array:

byte value in array = 255

In the file you write "255," which is 4 bytes - ASC("2"c), ASC("5"c), ASC("5"c), ASC(","c)

The binary serializer just writes the bytes, and it can load it back directly as an array as well...

Imports System.Runtime.Serialization.Formatters.Binary

Public Class Form1

Private Shared rand As New Random
Private saveName As String = "C:\test.bin" ' extension is irrelevant.
Private WithEvents b1 As New Button, b2 As New Button

Sub New()
InitializeComponent()
Me.Controls.AddRange(New Control() {b1, b2})
b1.Text =
"Save"
b2.Text = "Load"
b2.Location = New Point(b1.Left, b1.Bottom + 5)
End Sub


Sub SaveArray(ByVal bytes As Byte(), ByVal filename As String)
Try
Dim fs As IO.Stream = IO.File.OpenWrite(filename)
Dim bf As New BinaryFormatter
' Serialize the Array to the Stream.
bf.Serialize(fs, bytes)
fs.Flush()
fs.Dispose()
Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
End Sub

Function LoadArray(ByVal filename As String) As Byte()
Dim bytes() As Byte = Nothing
Try
Dim fs As IO.Stream = IO.File.OpenRead(filename)
Dim bf As New BinaryFormatter
' Deserialize the Stream, which creates an Object.
' Cast the Object to Byte().
bytes = DirectCast(bf.Deserialize(fs), Byte())
fs.Dispose()
Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
Return bytes
End Function

Private Sub b1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles b1.Click
Dim bytes(65535) As Byte
rand.NextBytes(bytes)
Console.WriteLine(
"The first few bytes before saving...")
For i As Integer = 0 To 2
Console.WriteLine(bytes(i).ToString)
Next
SaveArray(bytes, saveName)
End Sub

Private Sub b2_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles b2.Click
Dim bytes() As Byte = LoadArray(saveName)
Console.WriteLine(
"The first few bytes after loading...")
For i As Integer = 0 To 2
Console.WriteLine(bytes(i).ToString)
Next
End Sub
End
Class





Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Wibs2

Tall Dude, you are a STAR!!

Your code (plus a minor tweek or two) did the whole thing in milliseconds, when the msdn 'official' method for parsing csv text files was so painfully slow.

The following code read in the same 64k of values file, parsed the entire file, converted all the numerical values (in string format) into the byte format of their numerical values (ie 196,230,163, became C4 E6 A3), and then wrote them all out to a binary file. I checked the file with a Hex editor, and it was perfect.

Congrats, have a beer, have three!! It's the weekend, hope you get reeeeeeeally lucky!!

:-))

Wibs

Public Class Form1
Dim look3() As Byte

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

Dim n As Integer
Dim look As String = System.IO.File.ReadAllText("c:\64kRandom.csv") 'read in the dummy file
Dim look2() As String = Split(look, ",")
ReDim look3(look2.Length - 1)

For x As Integer = 0 To look3.Length - 1

n = CByte(Convert.ToInt32(look2(x)))
look3(x) = n ' update the byte array
ProgressBar1.Value = x / 655.35
Next
MsgBox("Done!")
My.Computer.FileSystem.WriteAllBytes("c:\csv.bin", look3, False)
End Sub

End Class






Re: Visual Basic Express Edition What on earth are 'Pumping Wait Primitives'? (CSV related)

Wibs2

Hi Jo0ls,

The exercise was to take a csv file, containing 65536 (256x256) values, in the form: 7,7,8,8,9,9, etc, and read it into a byte array.

Now it is in a byte array I will convert it into an image that can be displayed in a PictureBox.

There are great many formats out there for heightfield data, all use 256x256 pixels though, the most common file formats used being:

*.bmp, 8-bit, greyscale;
*.raw, (single channel) pixel data only. Imagine the pixel data in a bmp file (so, minus all the header, color table etc, and inverted to the right way up)
*.raw, special (13 channel version) of the raw file used by Second Life
*.csv, text file version of the single channel raw file, but all values on one line, no spaces, no ""

My app needed to read in any of these file types, and display the image in a PictureBox. The csv was the last format that needed doing.

Many thanks for your continued interest. I have learnt so much, and that there's more than one way to skin a VB cat!