codetale

i am trying to create my own game data file,
i want to pack all files/resources into a single file, which like "zip"/"cab" does.
but i have no idea how to implement this

i think the file structure should as below
<file_heaher><file_version>
<resources_list>
<resource_binary>
<file_footer>

  • file_header, use to reconize file's type
  • file_version, use to reconize file version, so that can use differ parser if required
  • resources_list, the list of files name
    so that my program can search for the embed files and locate where the file are stored and the file size
  • resource_binary, queue of file binary
my problem was
  1. how to record the each file size/information
  2. use what as separator
    in plain text, i can just simply use "linebrake" like
    this_is_file_a
    this_is_file_b
    ...
    or "comma"
    this_is_file_a,this_is_file_b,...
    but in binary, how to
is there any guide/articles

thank you


Re: Game Technologies: General File Data structure design

waruwaru

There are a couple of ways of doing this, you may want to think about what you want your file format to achieve. Do you always read the entire fire in a single shot, or would you like to be able to randomly access any one of the resource file at any time Do you care about speed Do you want to exchange the file between different platforms Lots of fun choices and lots of different way to approach this, for example:

- take a look at something like the iff format (or gif/jpg/zip..etc). You can keep the size info as part of per resource header. Since you know the how big your data content is, you don't need a separator.

- if you want separator, either pick something that will not appear in your data (maybe 0xDEADBEEF ). Or you will have to "escape" your data, like how you can use '/' in your strings. But this usually means you have to look at the bytes to figure out if you hit a separator

- You can keep your data in something like an XML format. You can uuencode your binary data into a string and keep it in a text file.

Btw, if you keep your data in binary and want to share between Windows/Xbox (maybe other platforms down the road), you have to be aware of endianness.






Re: Game Technologies: General File Data structure design

codetale

thankyou waruwaru

in my case, i want to able to randomly access any one of the resource file at any time with acceptable speed.
and the target platform are windows only.

i still not understand...
if the "resources info" without separator, then how i chunk them and locate the binary data

as my example before which are my idea

<file_heaher><file_version>
<resources_list>
<resource_binary>
<file_footer>


for the reouces_list, it should record
file_name, start_byte, file_size
so that i can read the correct binary from correct position and correct length, am i right

actually how the ".zip" store the each file information
i cant find any document on that




Re: Game Technologies: General File Data structure design

waruwaru

codetale wrote:
as my example before which are my idea

<file_heaher><file_version>
<resources_list>
<resource_binary>
<file_footer>


for the reouces_list, it should record
file_name, start_byte, file_size
so that i can read the correct binary from correct position and correct length, am i right

actually how the ".zip" store the each file information
i cant find any document on that

You got it. So if you keep track of the binary file size, then you know how much to read. For example, if your file looks something like a big array of resources:

<file_heaher><file_version>
<file_name><start_byte><file_size>
<resource_binary>
<file_footer>
<file_heaher><file_version>
<file_name><start_byte><file_size>
<resource_binary>
<file_footer>
<file_heaher><file_version>
<file_name><start_byte><file_size>
<resource_binary>
<file_footer>
...

You know how many bytes to read as file_header, and how many bytes to read as file_version, you should be able to read resource_list (file_name either fixed size, or terminated by null. constant size for start_byte, constant size for file_size). Now if you read in number of bytes stored in file_size, you got your resource binary, and your file pointer will be pointing at file_footer. After you read in file_footer, your file pointer will point to the next resource element. The problem with the approach is that you have to parse through the whole file before figuring out how many resources you have and where each resources are located.

You can find some info on the pkware zip file format here. There are alot of compression related data. But the basic idea is to keep a table of content structure that is easy to fine (at the end/beginning of file). So your program would read this structure first, and be able to quickly figure out the number of resources available, and offset where each of the resource is located. Which means you don't have to go through the entire file to figure out where resources are located within the file (as you have to do with the array of resource approach).






Re: Game Technologies: General File Data structure design

codetale

thank you waruwaru

the pkware zip file format will take times to read through.


the solution above you giving me, i guess that not working for me.
because i what to be able to get the resources at any time when required.
with you solution, it have to read all before i can using it, because the index are mixed with the binary.




Re: Game Technologies: General File Data structure design

waruwaru

Yep, so you can add a table of content structure at the beginning of your file, something like this would allow you to jump straigth to the resource:

<table_of_content>
<num_of_resources>
<resource_1_name><resource_1_offset><resource_1_size>
<resource_2_name><resource_2_offset><resource_2_size>
...
<resource_X_name><resource_X_offset><resource_X_size>
<end_of_toc>
<resource_1>
<resource_2>
...
<resource_X>

You only need either the offset or size, knowing size and where resource1 starts, you can calculate the offset for any resource. With offset, you can find the size of resource.