Often on field crews these days it is necessary to send large files of various sorts. They could be Field Stacks, Raw Field Records, Photographs, Reports or any number of other files.
The problem is that whilst the Oil Company (or Contractor) demand these files, they are seldom prepared to provide a decent VSAT link to allow it.
Recently I was in a situation of having to upload a 60MB field stack to an FTP site over a crew's shared 64KbS VSAT link. 64KbS is little better than a dial up modem. And of course being shared with everyone on crew meant even less was available to me for my big upload.
It is usually impossible to upload a file of this size to even an FTP site as the link just drops out. And it is too big for Email. So what is the answer?
My solution was to split the file into small chunks and send the chunks as separate files. Whoever downloads these chunks has to be able to join them all up of course.
There is an excellent piece of freeware available that does all this, it's called HJSplit and is available for download from here :
http://www.freebyte.com/hjsplit/
It's very easy to use, and handles both splitting and joining, so your recipient should have it also.
I usually split big files into 1MB (1000KB) chunks for uploading.
Splitting the file though is not the end of the story. How do you know that one of the chunks wasn't corrupted? You could (and probably should) ZIP the file before splitting - even a SEGY file will compress a little bit, and the ZIP procedure adds a CRC to the file so that when the rejoined file is Unzipped, if there is a corrupted chunk, then the recipient will know.
The trouble is, how do you determine which chunk is bad? There isn't a lot of use going to all the trouble of zipping/splitting/joining if the final result at the other end is still an unusable file!.
The answer here is to generate a Checksum or CRC for each of the chunks and have the recipient compare his with yours.
Whilst HJsplit allows you to generate a checksum for each file, it is very tedious to do 60 as in the case of my 60MB stack. Not only that a checksum is not guaranteed to catch multiple errors.
This prompted me to search for a program that would generate file CRCs for all files in a directory. Since I couldn't find what I was looking for, I wrote my own (sort of **).
The problem is that whilst the Oil Company (or Contractor) demand these files, they are seldom prepared to provide a decent VSAT link to allow it.
Recently I was in a situation of having to upload a 60MB field stack to an FTP site over a crew's shared 64KbS VSAT link. 64KbS is little better than a dial up modem. And of course being shared with everyone on crew meant even less was available to me for my big upload.
It is usually impossible to upload a file of this size to even an FTP site as the link just drops out. And it is too big for Email. So what is the answer?
My solution was to split the file into small chunks and send the chunks as separate files. Whoever downloads these chunks has to be able to join them all up of course.
There is an excellent piece of freeware available that does all this, it's called HJSplit and is available for download from here :
http://www.freebyte.com/hjsplit/
It's very easy to use, and handles both splitting and joining, so your recipient should have it also.
I usually split big files into 1MB (1000KB) chunks for uploading.
Splitting the file though is not the end of the story. How do you know that one of the chunks wasn't corrupted? You could (and probably should) ZIP the file before splitting - even a SEGY file will compress a little bit, and the ZIP procedure adds a CRC to the file so that when the rejoined file is Unzipped, if there is a corrupted chunk, then the recipient will know.
The trouble is, how do you determine which chunk is bad? There isn't a lot of use going to all the trouble of zipping/splitting/joining if the final result at the other end is still an unusable file!.
The answer here is to generate a Checksum or CRC for each of the chunks and have the recipient compare his with yours.
Whilst HJsplit allows you to generate a checksum for each file, it is very tedious to do 60 as in the case of my 60MB stack. Not only that a checksum is not guaranteed to catch multiple errors.
This prompted me to search for a program that would generate file CRCs for all files in a directory. Since I couldn't find what I was looking for, I wrote my own (sort of **).
Say for example we have split a file called
Final Stack Maysabongo L37.zip into 55 pieces.
We simply start The CRC Generator and point it to the folder that contains the 55 pieces:
Final Stack Maysabongo L37.zip into 55 pieces.
We simply start The CRC Generator and point it to the folder that contains the 55 pieces:
Select the FOLDER (in this case it is called Split) and Click OK
The CRCs for each file will be calculated and as soon as they are ready will be displayed in a text file:
The CRCs for each file will be calculated and as soon as they are ready will be displayed in a text file:
This file is always called File CRCs.TXT and is always created in the same folder as the files it is working on.
When you close the txt file, you can exit the CRC Generator - it will show 'Done'
When you close the txt file, you can exit the CRC Generator - it will show 'Done'
Now you can upload the 55x1MB files + File CRCs.TXT and let the recipient know that they are available.
The recipient downloads and joins all the files together.
If all is well, he can unzip the joined file and have the data set as it was originally. But if there is an error in the Unzip process like this:
The recipient downloads and joins all the files together.
If all is well, he can unzip the joined file and have the data set as it was originally. But if there is an error in the Unzip process like this:
Then we need to find out which of the 55 files send were bad. All the recipient needs to do is run the CRC Generator and point it at the folder that contains the downloaded chunks. The program will calculate CRCs and if the file File CRCs.TXT is found in the same directory, then it asks if a compare is required:
Click Yes and the program will indicate which files are good and which are bad:
In this case we see file 4 is bad, so the recipient can notify you to send file 4 and only file 4 again. It can save a lot of time and frustration.
** Earlier I said I wrote my own CRC Generator (sort of). Whilst I wish I could claim credit for the code that actually generates the CRCs it would not be true. All I did was make a 'wrapper' around it so that it is easy to use.
The actual CRC generation code was written by someone on the Powerbasic staff. Unfortuneatly the original post seems to have dissappeared from their forum so I cannot credit them directly, but for those interested, there is a host of discussions surrounding CRC32 that can be found here:
http://www.powerbasic.com/support/pbforums/
Just do a search for CRC32
My program just puts this code to use in a convenient way.
The actual CRC generation code was written by someone on the Powerbasic staff. Unfortuneatly the original post seems to have dissappeared from their forum so I cannot credit them directly, but for those interested, there is a host of discussions surrounding CRC32 that can be found here:
http://www.powerbasic.com/support/pbforums/
Just do a search for CRC32
My program just puts this code to use in a convenient way.