Hullo - Interested in Charas.EX

[starfox]

First, I'd like to say hullo and thanks for hosting this site! I'm excited about getting into sprites with the charaset generator.

Which brings me to a question: I see Charas.EX hasn't been updated in forever. I grabbed the latest and was disappointed to see that I couldn't update the art from beyond 2011 or thereabouts. Alas, such things happen when updated by volunteers. So I started trying to look at the format to see if I could maybe do a screen-scrape and then feed it back in. However, I noticed that everything was in a proprietary CMR format. After digging in the source code (woohoo! ...Purebasic? first time I've seen that...) the actual internal format looks not too hard (but I don't understand all the fields quite yet), BUT additionally PureBasic does this "OpenPack" thing such that the actual data appears to be compressed using some PureBasic specific format... makes it harder to work with. Out of this I have a couple questions:
-How were the original updates made? Did they use some internal format not available for download from this site? The CMR format made me wonder; I would definitely be curious in getting to the raw data.
-Is it possible for a new update to get created? It doesn't look like the maintainers are active for the Charas.EX project so I'm not hopeful.

And/or a really crazy idea:
In general I'm trying to think about the curation of the data here. You guys have a really good library here and I'd hate to see it ever go down with the site if it ever stops being hosted. Right now the logic all lives in PHP server-side and images are fetched one-by-one (which may be part of the whole server load issue). I did a rough screen scrape today and it looks like the whole works of Charas (for the charasets) is something less than 1.5MB (does that sound about right?). Since the hosting of this site started, HTML Canvas has really taken off and I wonder if it wouldn't be possible to 1) push the resources into a single heavy pack and 2) do all processing in JS...

Whew...! That was a whole load of crazy ideas. Thoughts?

[starfox]

FYI, I've had a go at parsing the Charas.EX format.
One roadblock for myself and others in the past is the nasty compression that PureBasic had in place; it appears that PureBasic has phased out the format as well so I had trouble getting the demo to decompress the CMR files for me as well. I can now open these in .NET.
The format relies on the JCALG1 compression method. An API - and importantly, a test program - can be found here: https://bitsum.com/other/
The format of the archive for the cmr file is then simply:
[CompressedBlock]*
[CompressedBlock] -> [U32 uncompressed size] [U32 compressed block size] [Raw JCALG1 data starting with 'JC']
Charas.EX appears to use archives with two files: the actual data file is first, then a preview file (BMP in the ones I've seen so far).

I've also had a go at that first data chunk, which I'll call "uncompressed CMR" for now. I need to validate this part before posting further, but basically it looks like a special 8 bit bitmap/palette combo with a few more goodies. (Source code with German variable names is interesting...)

[Momeka]

 I think the whole upload system is broken and the people who can answer your questions are long gone.

[Fisherson]

First, I'd like to say hullo and thanks for hosting this site! I'm excited about getting into sprites with the charaset generator.

Which brings me to a question: I see Charas.EX hasn't been updated in forever. I grabbed the latest and was disappointed to see that I couldn't update the art from beyond 2011 or thereabouts. Alas, such things happen when updated by volunteers. So I started trying to look at the format to see if I could maybe do a screen-scrape and then feed it back in. However, I noticed that everything was in a proprietary CMR format. After digging in the source code (woohoo! ...Purebasic? first time I've seen that...) the actual internal format looks not too hard (but I don't understand all the fields quite yet), BUT additionally PureBasic does this "OpenPack" thing such that the actual data appears to be compressed using some PureBasic specific format... makes it harder to work with. Out of this I have a couple questions:
-How were the original updates made? Did they use some internal format not available for download from this site? The CMR format made me wonder; I would definitely be curious in getting to the raw data.
-Is it possible for a new update to get created? It doesn't look like the maintainers are active for the Charas.EX project so I'm not hopeful.

And/or a really crazy idea:
In general I'm trying to think about the curation of the data here. You guys have a really good library here and I'd hate to see it ever go down with the site if it ever stops being hosted. Right now the logic all lives in PHP server-side and images are fetched one-by-one (which may be part of the whole server load issue). I did a rough screen scrape today and it looks like the whole works of Charas (for the charasets) is something less than 1.5MB (does that sound about right?). Since the hosting of this site started, HTML Canvas has really taken off and I wonder if it wouldn't be possible to 1) push the resources into a single heavy pack and 2) do all processing in JS...

Whew...! That was a whole load of crazy ideas. Thoughts?

@-@ All that code speech hurts my brain. I modified the contents of the Charas Ex a while back but I couldn't tell you how to pack it up and make it downloadable. ^^;

[starfox]

Well, can anybody tell me what the "raw" backend format is and how to get the data? I get the feeling from Charas.EX that it's not just the PNG but a couple other details, and that's (maybe?) how they made the updates... Otherwise the best shot I've got is a screen scrape of the pngs vs. the transparent pngs to mask out the background to try to get CMRs created... Yikes! Any nuggets 'o info would be greatly appreciated.

...now to navigate the forum post purgatory of captcha!

[starfox]

And Momeka, do I understand you to mean that upload's broken for everyone? I couldn't log in to the Charas generator, let alone upload, but I wasn't just sure if it was me...

[Momeka]

Well, can anybody tell me what the "raw" backend format is and how to get the data? I get the feeling from Charas.EX that it's not just the PNG but a couple other details, and that's (maybe?) how they made the updates... Otherwise the best shot I've got is a screen scrape of the pngs vs. the transparent pngs to mask out the background to try to get CMRs created... Yikes! Any nuggets 'o info would be greatly appreciated.

...now to navigate the forum post purgatory of captcha!

I'm pretty sure you just uploaded a png back in the day but no idea what happens back end after that.

And Momeka, do I understand you to mean that upload's broken for everyone? I couldn't log in to the Charas generator, let alone upload, but I wasn't just sure if it was me...

It's broken for everyone, been that way for years.

[Prpl_Mage]

...now to navigate the forum post purgatory of captcha!

Hang in there, it won't last forever, just our desperate anti-bot spam meassure

[Fisherson]

So somethign weird happened. My PC has been experieincing this random error, which my brother got rid of but till he did I thought I needed to back stuff up. So I went and backed up Charas Ex but somehow broke the modified one I did.  < <; Irritiating but anyway as I was unpacking it from what I thought was the original file and thinking I'd have to import all my resources again? It turns out this is that crazy experimental file with my imported stuff on it and what's crazier? It worked! O_O I mean I love the huge fap-inducing body set I had but allot of the other stuff is still there. From a rar file I made myself. >.> So , stary mah boy, it seems all you have to do to update the sets, is import new sets and make sure they match up and have no errors then simply pack it away in win rar. o.o If I had known that would could have updated the program AGES ago! XD In fact if somebody will format hair,clothes and such I'll import it in mine if you ya'll want.

[starfox]

Indeed - the plot thickens!

So last night I went nuts and downloaded all the charasets automagically and updated/fixed transparency so now I have the current set with "normal" png transparency. My general thought is if I could get these, in this format, maybe two things would be possible:

1) The data would be very hackable. While I like Charas.EX, that CMR format is going to make it a bear to work with if anybody wants to do something more off the beaten path like algorithmically create sprites... Additionally, the new style PNG transparency seems to me to have some advantages, but... how will the older RPG makers play with that....? I'm planning on coding something in C++ with SFML directly so I have a lot more flexibility.

2) There was once upon a time talk of a client-side heavy client rather than doing all the things in PHP... is this still something the community might want...? I'm thinking all sorts of crazy ideas like maybe a GitHub hosted page that has the resources and a simple online/offline sprite maker available. That way the sprites can be offloaded from this site if the traffic is still an issue and additionally the repo can be replicated, forked etc. for everybody trying to hack on the images.

I dunno. Just not sure where to go from here from a community-sharing perspective. Keep in mind I'm a bit of a noob to this stuff... But Fisherson, I sure can ship you what I've got at some point if you're curious; my original estimate was wrong and it's about 11MB.

[Fisherson]

I dunno. Just not sure where to go from here from a community-sharing perspective. Keep in mind I'm a bit of a noob to this stuff... But Fisherson, I sure can ship you what I've got at some point if you're curious; my original estimate was wrong and it's about 11MB.

Sure I can take a look at it. ^_^ Though half of what you say goes over my head. XD I'm not a coder but I know how to import new clothes and accessories and stuff to it. Been doing that for years. Just got lazy and left broken ones in and didn't adjust some others that work in color scheme but not many. =/ Either way sure why not?

[starfox]

Ok - sent you a PM.