Error Code Modeler compression

[emulover]

Hi everyone and pSX author

Niel corlett has made a Error Code Modeler.

using this tool cd isos's can be compressed much better.

on average files are compressed 20% better.

a guy called x243x243 has made an automated tool that will 7z or rar files using the error code modeler.

I was wondering is this system already being used in cdz compression?

I would think that ecd is also used in psp psone isos

It would be great if the ECM'd 7zipped files could be supported

links are here
www.neillcorlett.com/ecm/
www.underground-gamer.com/forums.php?action=viewtopic&topicid=5212


Edit:

I tested Silent Hill.bin (usa image)
.bin: 602,046KB
.rar: 305,667KB
.cdz: 332.078
.ECM7z: 256,959KB
.ECMRAR: 271.196

[Ultima]

The usual compression algorithms do not lend themselves to data seeking, so 7-zip'd files (as well as regular RAR, ZIP, ACE, etc. files) can't be supported.

As for ECM itself... The reference code is GPL'd, so unless there is a detailed description/specification of how the algorithm works, I don't think pSX Author can make much use of it -- he can't exactly use the code, and I personally am not sure whether the "How it works" page" is detailed enough (perhaps it might be).

Additionally, I doubt there's a way to convert from ECM back to the regular image format quickly and on-the-fly (converting back is required before you can use the image), so I doubt this is of much use with CDZ.

[emulover]

I am affraid you are correct.

But im still hoping for some kind of miracle, at the very lease ECM7z can be used as a backup format

[Ultima]

pSX itself has been a miracle of sorts, so I agree with you -- we can always hope

[f1reb4ll]

ECM doesn't compress anything, it cuts out ECC and EDC fields from the image.

[Ultima]

Indeed, but emulover was referring to ECM'd, then compressed images.

[states]

it only cuts out the redundant edc and ecc fields, in other words the repeated ones, at least that is what I was lead to believe when I read the documentation that came with it...

[Ultima]

Nope. While I'm no expert on the subject (lol, I know next to nothing about CD file systems, truthfully), what I got out of reading the documentation was that all ECC and EDC data are cut out if they're valid for the disc data (junk ECC/EDC data aren't tossed in case it's intentionally invalid). ECC and EDC, IINM, can easily be generated again, which is part of the process of converting ECM'd images back to its native format.

Edit: Additionally, the removal of repeat data wouldn't help make the compression ratio better for the usual compression algorithms, as they already check for redundant data while compressing.

[emulover]

EDC and ECC are by definition uncompressable because they appear as random garbage data to the compressor, thos removing carefully selected and rebuildable EDC and ECC can result in a file that is many many times more smaller than the original(when compressed). Uncompressed ECM'd isos are only about 50 mb smaller

[monkey80]

it also works with mdf but make sure when you compress the cd image with the mds and finally keep the cue for some bin images that have audio tracks which you can easily check by opening a cue file with your text editors.

I name my ecm like below
mgs(bin).ecm (so I know the original is in bin format).
mgs.cue is kept with the mgs(bin).ecm and kept in a compress files.

[Sotho Tal Ker]

Using ECM is similar to using the iso format with it's 2048bytes block size.

The ECC and EDC data is usually completely random for the compressor, so by eliminating it you can make the image compress much better.

The problem lies in the GPL license - at least for closed source PSX emulators.
An option would be to code it as an extra DLL where the source code can be given out on request seperated from the main pSX source. (Remember, you only have to provide the source ON request, the GPL does not force you to give out the code with every release)

From the GNU website

Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it

The code for ECM and UNECM is not that big, and a skilled programmer can surely use it as an example for coding his own version.

ECM/UNECM could be integrated into CDZ compression itself.. first removing the ECM/ECC data, then compressing the stuff using the CDZ compression algorithm.

Sadly, pSXAuthor did not yet post the specifications of the CDZ algorithm yet, except this:

CDZ uses zlib, but its not just straight zlib compression. It is designed for random access (which zlib/etc.. do not support well). So the main binary section is compressed in "chunks" with a directory stored at the end. CDZ file also contain various other compressed stuff at the start (eg. the .cue or .ccd file).

[Ultima]

Whether it could be added to CDZ compression wasn't really the problem at hand. The problem is the fact that CDZ and ECM have different goals. CDZ is used for compression with on-the-fly decompression. ECM is for improved compression ratios where on-the-fly decompression isn't even a question, since it's used mainly for archiving purposes only.