Introducing RawSpeed
I have spent quite a lot of my spare time in the last two months creating a file loader for RAW files. My primary motivation, beside general interest was, that I was frustrated there wasn't any fast open source loaders out there.
So my main objectives were to make a very fast loader that worked for 75% of the cameras out there, and was able to decode a RAW file at close to the optimal speed. The last 25% of the cameras out there could be serviced by a more generic loader, or convert their images to DNG - which as a sidenote usually compresses better than your camera.
So far, I have support of the latest cameras from Canon, Nikon, Sony, Pentax and Olympus, besides DNG support. I plan to add Panasonic cameras to the mix, but that's about it. RawSpeed has been integrated into the upcoming version of Rawstudio, which is being maintained by my friend Anders Brander.
My natural choice of language was C++, and while I considered adding assembler, I haven't seem any obvious spots where it would help more than a few percent. That would also make it fairly easy to port, as a side-bonus. My C++ style is "object-oriented C", with carefully chosen STL use, inspired by AviSynth, so I knew it wouldn't hinder performance much.
Compared to other raw decoders out there, I decided from the start to completely forego file streams. The image must be fully loaded into memory before decoding can start. This would first of all avoid many system calls, non-serial IO, and futhermore makes it possible to have threaded IO and decoding, by having an IO thread.
I feel like I must mention the inspiration. Obviously the closest relative is libopenraw. It seems like a nice project. I considered contributing to the project, but since I had already made a TIFF parser, and making it "streamless" would make it almost a complete rewrite, I decided I might as well write my own. At least then I cannot blame bugs on other people.
The other wellknown project out there is Dave Coffin's dcraw. David has put an incredible amount of work into reverse-engineering the strangest formats out there and creating a solid application that works for jsut about any camera out there. While he has a radically different coding style than most of what you find out there, the amount of work and dedication put into this project is very admirably.
So far the only thing you can see it the Subversion repository, but I will make a formal release, when the time is right. If you want to follow the development, you can subscribe to the rawstudio commit mailing list, where my updates are also posted.
That's it for now. Next time I hope to be able to tell you a bit about what the library actually contains.
So my main objectives were to make a very fast loader that worked for 75% of the cameras out there, and was able to decode a RAW file at close to the optimal speed. The last 25% of the cameras out there could be serviced by a more generic loader, or convert their images to DNG - which as a sidenote usually compresses better than your camera.
So far, I have support of the latest cameras from Canon, Nikon, Sony, Pentax and Olympus, besides DNG support. I plan to add Panasonic cameras to the mix, but that's about it. RawSpeed has been integrated into the upcoming version of Rawstudio, which is being maintained by my friend Anders Brander.
My natural choice of language was C++, and while I considered adding assembler, I haven't seem any obvious spots where it would help more than a few percent. That would also make it fairly easy to port, as a side-bonus. My C++ style is "object-oriented C", with carefully chosen STL use, inspired by AviSynth, so I knew it wouldn't hinder performance much.
Compared to other raw decoders out there, I decided from the start to completely forego file streams. The image must be fully loaded into memory before decoding can start. This would first of all avoid many system calls, non-serial IO, and futhermore makes it possible to have threaded IO and decoding, by having an IO thread.
I feel like I must mention the inspiration. Obviously the closest relative is libopenraw. It seems like a nice project. I considered contributing to the project, but since I had already made a TIFF parser, and making it "streamless" would make it almost a complete rewrite, I decided I might as well write my own. At least then I cannot blame bugs on other people.
The other wellknown project out there is Dave Coffin's dcraw. David has put an incredible amount of work into reverse-engineering the strangest formats out there and creating a solid application that works for jsut about any camera out there. While he has a radically different coding style than most of what you find out there, the amount of work and dedication put into this project is very admirably.
So far the only thing you can see it the Subversion repository, but I will make a formal release, when the time is right. If you want to follow the development, you can subscribe to the rawstudio commit mailing list, where my updates are also posted.
That's it for now. Next time I hope to be able to tell you a bit about what the library actually contains.
9 Comments:
I think the most gain could be made from using something like openMP to take advantage of multiple cores. Most new computers are dualcore, and someone into computer graphics may easily have a quad.
Decoding an image is surely something that is easy to make parallel.
By
Anonymous, at 2:42 pm
@Anonymous: DNG is already multithreaded, using pthreads, and the speedup is close to optimal (3.5x+ on a Quadcore).
Most other formats, CR2 as an example must be completely linear decoded, since every pixel/line is depending on the previous one. That said, CR2 decodes at about 39 Megapixels/second, so all files should be within a second.
DNG is decoding at ~100 Megapixels per second on a quadcore, since the speedup of a quadcore can be utilized fully.
By
Klaus Post, at 3:09 pm
Why didn't you just discuss the matter with me for libopenraw. I'm pretty open and for once I was not getting the "I don't know/want C++".
By
Hub, at 2:48 am
@hub: Wrote you an email :)
By
Klaus Post, at 11:59 am
great new library, very clean and really speedy, too!
two things:
- is there a simple way to access camera white balance data from rawspeed?
- i had to patch FileReader.cpp to avoid infinite loops when trying to open a directory (silly use case, i know):
@@ -56,7 +60,13 @@
while ((st.st_size > 0) && (bytes_read < st.st_size)) {
dest = (char *) fileData->getDataWrt(bytes_read);
- bytes_read += read(fd, dest, st.st_size - bytes_read);
+ ssize_t ret = read(fd, dest, st.st_size - bytes_read);
+ if(ret < 0)
+ {
+ perror("read");
+ throw FileIOException("Could not open file.");
+ }
+ bytes_read += ret;
}
close(fd);
cheers,
jo
By
Anonymous, at 4:11 pm
@jo:
>>is there a simple way to access camera white balance data from rawspeed?
Whitebalance is not read by Rawspeed at all. Here at the beginning, I have only focussed on decoding the raw data itself with correct crop/CFA pattern.
I don't know if exiv2 would help you greatly there. For Rawstudio there is a separate meta-data reader:
http://rawstudio.org/svn/rawstudio/trunk/plugins/meta-tiff/tiff-meta.c
Regarding your patch, that loop has actually been re-written, since I had to change file size detection.
I have not had the time to turn RawSpeed into a "proper" library yet. You are of course very welcome to contact me by mail, or add bugs on rawspeed in the Rawstudio bug tracker.
By
Klaus Post, at 9:40 am
Are there any plans to release rawspeed as a full library or integrate it into libopenraw?
Is it possible to read raw files at half or quarter resolution? Would that be a further speed boost in reading (and later processing)?
By
Pedro Côrte-Real, at 1:31 am
I have successfully checked out rawspeed from svn:
"svn co https://rawstudio.org/svn/rawstudio/trunk/ rawspeed"
./autogen.sh
make
sudo make install
But I'm lacking a bit on documentation. Where can I find some? Does rawspeed operate only as a plugin to rawstudio or is there a standalone program like dcraw? If so where did it get installed (I expected a "rawspeed" executable installed somewhere in my path).
By
Unknown, at 3:24 am
Lovely bllog you have
By
Sidney, at 8:46 am
Post a Comment
<< Home