Torgie

Hello, all. I've pored over the MSDN documentation on File I/O, and done some searching here in the Forums, but can't find an answer to this particular problem I've encountered!

The project I'm working on requires a very sensitive approach to file reads and writes. It will be working with very large files, upwards of 2gB, large enough that a ReadAllLines() call is out of the question. It also works with removeable storage, namely flash drives, so I can't go around making copies of these giant files either. What I'm trying to do is trim/remove characters from a file while keeping it "in-place".

Here's a concrete example. Say I have an text file that's 256mB sitting on a thumb drive which is itself 256mB. I'm also not allowed to write anything to the disk - the flash drive is effectively the only thing that has write access. A user has performed a task and I want to remove the first 64 bytes from the top of that text file.

Code Block


// fp is a FileStream which has, in this example, read 64 bytes

// I reset the position to the beginning of the read

fp.Position = fp.Position - bytesRead;




// And right now I'm just writing nulls to the file to "scrub" the expired characters
for(int i = 0; i < bytesRead; i++)
fp.WriteByte(0);


// But then when I read the file back in later,
// I have to wait while it finds the start of the next read!
for(long position = 0; true; position++)
if(fp.ReadByte() != 0)
{
fp.Position = position;
break;
}



My "solution" is a hack - the for(true)/break makes me cringe. There has to be a way to chop off those leading characters without loading the entire thing into memory or making copies of the file! Any help would be wonderful.
-torgie


Re: .NET Base Class Library Remove bytes from the top of a file

JohnWein

Reserve the first 8 bytes in the file for a long containing the first position in the file.





Re: .NET Base Class Library Remove bytes from the top of a file

Torgie

[Edit] Hey, did this post get moved into .NET Base Class Library I posted it in C# General! [/Edit]

Ah yes, that would work - but for security reasons, once bytes have been read from the file they must be removed. As you might have guessed, I'm working in matters related to cryptography

I guess I could combine my "null-bytes overwrite" method with the "first-8-bytes represent the pointer". That would get rid of the giant pause when opening a file with a gig of null bytes at the beginning... it would also get rid of the for/break madness. I'll go ahead and implement that work-around! Thanks!

Academically, though, it still seems like there should be a way to remove bytes from a file without copying it or loading the entire thing into memory at once! Does anyone know of a way




Re: .NET Base Class Library Remove bytes from the top of a file

Centuri0n

File stored on disc "as is", means if you remove some bytes from start you have to completely rewrite all file.

File breaked into sectors in fat system, so if you need to remove completely one custom sector of file, be that first sector, last or somewhere in the middle this can be done faster just adjusting fat table, so workaround you looking for can be only on low-level. This is only in case bytes to remove equals sector size, and probably will need you some extra library. So this is really hard way, i guess it's much easier to add pointer, or break file into ten millions parts