CDDA Paranoia FAQ ---------------------------------------------------------------------------- "Suspicion Breeds Confidence!" --Brazil ---------------------------------------------------------------------------- August 20, 1999 For those new to Paranoia and cdparanoia, this is the best, first place to look for information and answers to your questions. More information can be found on the cdparanoia homepage: http://www.xiph.org/paranoia/ ---------------------------------------------------------------------------- Table of Contents 1. Questions about the Paranoia and cdparanoia projects 1. What is cdparanoia? 2. Why use cdparanoia? 3. What is Paranoia? 4. Is cdparanoia / Paranoia portable? 5. What is Paranoia's history? 6. Is cdparanoia/Paranoia related to cdda2wav? 7. What are the differences between Paranoia II, III and IV? 8. Are there cdparanoia mailing lists for users or developers? 9. What is Paranoia IV's current development status? 10. Will cdparanoia, and cdda2wav or xcdroast merge anytime in the future? 2. Questions about using Paranoia and cdparanoia 1. Requirements to run cdparanoia (as of alpha 3) 2. Does Cdparanoia support ATAPI drives? SCSI Emulation? Parallel port drives? 3. I can play audio CDs perfectly; why is reading the CD into a file so difficult and prone to errors? 4. Does cdparanoia lose quality from the CD recording? 5. Can cdparanoia detect pregaps? Can it remove the two second gaps between tracks? 6. Why don't you implement CDDB? A GUI? Four million other features I want? 7. The progress meter: What is that weird bargraph during ripping? 8. How can I tell if my drive would be OK with regular cdda2wav? 9. What is the biggest value of SG_BIG_BUFF I can use? 10. Why do the binary files from two reads differ when compared? 11. Why does CDParanoia rip files off into WAV format (and other sample formats) but not CDDA format? ------------------------------------------------------------------------- Questions about the Paranoia and cdparanoia projects What is cdparanoia? Cdparanoia is a Compact Disc Digital Audio (CDDA) extraction tool, commonly known on the net as a 'ripper'. The application is built on top of the Paranoia library, which is doing the real work (the Paranoia source is included in the cdparanoia source distribution). Like the original cdda2wav, cdparanoia package reads audio from the CDROM directly as data, with no analog step between, and writes the data to a file or pipe in WAV, AIFC or raw 16 bit linear PCM. Cdparanoia is a bit different than most other CDDA extration tools. It contains few-to-no 'extra' features, concentrating only on the ripping process and knowing as much as possible about the hardware performing it. Cdparanoia will read correct, rock-solid audio data from inexpensive drives prone to misalignment, frame jitter and loss of streaming during atomic reads. Cdparanoia will also read and repair data from CDs that have been damaged in some way. At the same time, however, cdparanoia turns out to be easy to use and administrate; It has no compile time configuration, happily autodetecting the CDROM, its type, its interface and other aspects of the ripping process at runtime. A single binary can serve the diverse hardware of the do-it-yourself computer laboratory from Hell... ----------------------------------------------------------------------- Why use cdparanoia? All CDROM drives are not created equal. You'll need cdparanoia if yours is a little less equal than others-- or maybe you just keep your CD collection in a box of full of gravel. Jewel cases are for wimps; you know what I'm talking about. Unfortunately, cdda2wav and readcdda cannot work properly with a large number of CDROM drives in the desktop world today. The most common problem is sporadic or regular clicks and pops in the read sample, regardless of 'nsector' or 'overlap' settings. Cdda2wav also cannot do anything about scratches (and they can cause cdda2wav to break). Cdparanoia is also smarter about probing CDDA support from SCSI and IDE-SCSI drives; many drives that do not work at all with cdda2wav, readcdda, tosha, etc, will work just fine with cdparanoia. ----------------------------------------------------------------------- What is Paranoia? Paranoia is a library project that provides a platform independent, unified, robust interface for packet-command based devices. In the case of CDROM drives for example, handling and programming cdrom drives becomes identical whether on Solaris or Linux, or if the Linux drive is SCSI, ATAPI or on the parallel port. In this way, Paranoia is similar to Joerg Schilling's SCG library. In addition to device/platform unification, the library provides tools for automatically identifying devices, and intelligent handling/correction of errors at all levels of the interface. On top of a generic low-level packet command layer, Paranoia implements high-level error-correcting interfaces for tasks such as CDDA where broken or vastly non-standard devices are the rule, rather than the exception. The Paranoia libraries are incomplete; the first release for use will be Paranoia IV, to be bundled with cdparanoia alpha release 10. Programming documentation for Paranoia IV will appear shortly on the documentation page as Programming with Paranoia IV. Programmers interested in contributing to Paranoia IV should read the heading Paranoia IV development information. ----------------------------------------------------------------------- Is cdparanoia / Paranoia portable? Paranoia III is Linux only (although it runs on all the flavors of linux with a 2.0 or later kernel. It is not only for x86). Paranoia IV (cdparanoia alpha 10 and later) is a port to other UNIX flavors and uses a substantially revised infrastructure. NetBSD and Solaris will be first; others will be added as time and outside assistance allow. Suggestions on the proper way to handle each OS's native configuration idioms are welcome. I want Rhapsody cdparanoia to look just like other Rhapsody apps just as much as I want Linux cdparanoia to look like a Linux app. ----------------------------------------------------------------------- What is Paranoia's history? Is cdparanoia/Paranoia related to cdda2wav? Paranoia I/II and cdparanoia began life as a set of patches to Heiko Eissfeldt's 'cdda2wav' application. Cdparanoia gained its own life as a rewrite of cdda2wav in January of 1998 as "Paranoia III". Paranoia III proved to have an inadequate structure for extention and use on other platforms, so Paranoia IV began to take form in fall of 1998. Modern Paranoia no longer has any relation to cdda2wav aside from general cooperation in sharing details between the two projects. In fact, cdda2wav itself doesn't look much like the cdda2wav of a year or two ago. ----------------------------------------------------------------------- What are the differences between Paranoia II, III and IV? Paranoia I and II were a set of patches to Heiko Eissfeldt's cdda2wav 0.8. These patches did nothing more than add some error checks to the standard cdda2wav. They were inefficient and only worked with some drives. Paranoia III was the first version to be written seperately from cdda2wav in the form of a standalone library. It was not terribly portable, however, and the API proved to be inadequate for extension. Paranoia IV is the upcoming new generation of CDDA Paranoia. It is both portable and more capable than Paranoia III. ----------------------------------------------------------------------- Are there cdparanoia mailing lists for users or developers? Yes. In addition to the mailing lists below, read-only CVS access to Paranoia III and IV will be availble from xiph.org soon (Paranoia IV is not yet under CVS). See http://www.xiph.org/paranoia/ for upto date information and automated ways of subscribing. Mailing list for Paranoia and Cdparanoia users (paranoia@xiph.org): To join: send a message containing only the one-word line 'subscribe' in the body to paranoia-request@xiph.org. Do not send subscription requests directly to the main list. The list server at xiph.org should respond fairly quickly with a welcome message. Mailing list for Paranoia IV developers: paranoia-dev@xiph.org The developers list is intended for focused development discussion amongst the core Paranoia development team and outside groups developing their own applications using Paranoia. Of course, anyone is welcome to read. To join: send a message containing only the one-word line 'subscribe' in the body to paranoia-dev-request@xiph.org. Do not send subscription requests directly to the main list. List for general CDROM tools There's also a general mailing list for those using/developing CDDA extraction and CD writing tools (cdwrite@other.debian.org). Subscribe by sending mail to other-cdwrite-request@lists.debian.org containing only the word subscribe in the body. Do not send subscription requests directly to the main list. ----------------------------------------------------------------------- What is Paranoia IV's current development status? Paranoia IV code will soon be available for internal evaluation, testing and development work to the developers involved in the Paranoia project; read-only CVS access should also be available soon. A public release does not yet set for any firm date. Those interested in contributing to the development of Paranoia, or who wich to contribute to porting to other platforms, please contact us. Paranoia IV prerelease code will be available to porters soon; I prefer to be in contact with those porting to other platforms so that Paranoia development has consistent quality across platforms. At the moment, volunteers have contacted me for most major platforms, but more help is still welcome on every OS. ----------------------------------------------------------------------- Will cdparanoia, and cdda2wav or xcdroast merge anytime in the future? Probably not beyond the point it already has. Versions of XCDRoast (and other GUI frontends; see the links page) that make use of cdparanoia already exist. Although the cdrecord/cdda2wav and Paranoia projects cooperate, they're likely to remain seperate as the former is committed to Joerg Schilling's libscg (part of the cdrecord package), just as cdparanoia is committed to using Paranoia IV. ----------------------------------------------------------------------- Questions about using Paranoia and cdparanoia Requirements to run cdparanoia (as of alpha 3) 1. A CDDA capable CDROM drive 2. Linux 2.0, 2.1, 2.2 or 2.3 1. kernel support for the particular CDROM in use 2. kernel support for the generic SCSI interface (if using a SCSI CDROM drive) and proper device (/dev/sg?) files (get them with the MAKEDEV script) in /dev. Most distributions already have the /dev/sg? files. The cdparanoia binary will likely work with Linux 1.2 and 1.3, but I do not actively support kernels older than 2.0 I do know for a fact that the source will not build on kernel installs older than 2.0, but the problems are mostly related to the ever-changing locations of proprietary cdrom include files. Also, although a 2.0 stock SCSI setup will work, performance will be better if linux/include/scsi/sg.h defines SG_BIG_BUFF to 65536 (it can't be bigger). Recent kernels (2.0.30+?) already set it to 32768; that's OK. Cdparanoia will tell you how big your generic SCSI buffer is. 2.2+ does not use a static DMA pool for SG, so there is nothing to tune. Unlike cdda2wav, cdparanoia does not require threading, IPC or (optionally) sound card support. /proc filesystem support is no longer required (but encouraged!), and /dev/sr? or /dev/scd? devices are not required for SCSI, although they do add functionality if present. ----------------------------------------------------------------------- Does Cdparanoia support ATAPI drives? SCSI Emulation? Parallel port drives? Alpha 9 supports the full ATAPI, IDE-SCSI and SCSI generic interfaces under Linux. Note that the native ATAPI driver is supported, but that IDE-SCSI emulation works better with ATAPI drives. This is an issue of control; the emulation interface gives cdparanoia complete control over the drive whereas the native ATAPI driver insists on hiding the device under an abstraction layer with poor error handling capabilities. Note also that a number of ATAPI drives that do not work at all with the ATAPI driver (error 006: Could not read audio) *will* work with IDE-SCSI emulation. Parallel port based CDROM (paride) drives are not yet supported; support for these drives in Linux will appear in alpha release 10 (Paranoia IV). ----------------------------------------------------------------------- I can play audio CDs perfectly; why is reading the CD into a file so difficult and prone to errors? It's just the same thing. Unfortunately, it isn't that easy. The audio CD is not a random access format. It can only be played from some starting point in sequence until it is done, like a vinyl LP. Unlike a data CD, there are no synchronization or positioning headers in the audio data (a CD, audio or data, uses 2352 byte sectors. In a data CD, 304 bytes of each sector is used for header, sync and error correction. An audio CD uses all 2352 bytes for data). The audio CD *does* have a continuous fragmented subchannel, but this is only good for seeking +/-1 second (or 75 sectors or ~176kB) of the desired area, as per the SCSI spec. When the CD is being played as audio, it is not only moving at 1x, the drive is keeping the media data rate (the spin speed) exactly locked to playback speed. Pick up a portable CD player while it's playing and rotate it 90 degrees. Chances are it will skip; you disturbed this delicate balance. In addition, a player is never distracted from what it's doing... it has nothing else taking up its time. Now add a non-realtime, (relatively) high-latency, multitasking kernel into the mess; it's like picking up the player and constantly shaking it. CDROM drives generally assume that any sort of DAE will be linear and throw a readahead buffer at the task. However, the OS is reading the data as broken up, seperated read requests. The drive is doing readahead buffering and attempting to store additional data as it comes in off media while it waits for the OS to get around to reading previous blocks. Seeing as how, at 36x, data is coming in at 6.2MB/second, and each read is only 13 sectors or ~30k (due to DMA restrictions), one has to get off 208 read requests a second, minimum without any interruption, to avoid skipping. A single swap to disc or flush of filesystem cache by the OS will generally result in loss of streaming, assuming the drive is working flawlessly. Oh, and virtually no PC on earth has that kind of I/O throughput; a Sun Enterprise server might, but a PC does not. Most don't come within a factor of five, assuming perfect realtime behavior. To keep piling on the difficulties, faster drives are often prone to vibration and alignment problems; some are total fiascos. They lose streaming *constantly* even without being interrupted. Philips determined 15 years ago that the CD could only be spun up to 50-60x until the physical CD (made of polycarbonate) would deform from centripetal force badly enough to become unreadable. Today's players are pushing physics to the limit. Few do so terribly reliably. Note that CD 'playback speed' is an excellent example of advertisers making numbers lie for them. A 36x cdrom is generally not spinning at 36x a normal drive's speed. As a 1x drive is adjusting velocity depending on the access's distance from the hub, a 36x drive is probably using a constant angular velocity across the whole surface such that it gets 36x max at the edge. Thus it's actually spinning slower, assuming the '36x' isn't a complete lie, as it is on some drives. Because audio discs have no headers in the data to assist in picking up where things got lost, most drives will just guess. This doesn't even *begin* to get into stupid firmware bugs. Even Plextors have occasionally had DAE bugs (although in every case, Plextor has fixed the bug *and* replaced/repaired drives for free). Cheaper drives are often complete basket cases. Rant Update (for those in the know): Several folks, through personal mail and on Usenet, have pointed out that audio discs do place absolute positioning information for (at least) nine out of every ten sectors into the Q subchannel, and that my original statement of +/-75 sectors above is wrong. I admit to it being misleading, so I'll try to clarify. The positioning data certainly is in subchannel Q; the point is moot however, for a couple of reasons. 1. The SCSI and ATAPI specs (there are a couple of each, pick one) don't give any way to retrieve the subchannel from a desired sector. The READ SUB-CHANNEL command will hand you Q all right, you just don't have any idea where exactly that Q came from. The command was intended for getting rough positioning information from audio discs that are paused or playing. This is audio; missing by several sectors is a tiny fraction of a second. 2. Older CDROM drives tended not to expect 'READ SUB-CHANNEL' unless the drive was playing audio; calling it during data reads could crash the drive and lock up the system. I had one of these drives (Apple 803i, actually a repackaged Sony CD-8003). 3. MMC-2 *does* give a way to retrieve the Q subchannel along with user data in the READ CD command. Although the drive is required to recognize the fetaure, it is allowed to simply return zeroes (effectively leaving the feature unimplemented). Guess how many drives actually implement this feature: not many. 4. Assuming you *can* get back the subchannel, most CDROM drives seem to understand audio discs primarily at the "little frame" level; thus sector-level structures aren't reliable. One might get a reassembled subQ, but if the read began in the middle of a sector (or dropped a little frame in the middle; many do), the subQ is likely corrupt and useless. As reassembling uncorrupted frames is easy without the subchannel, and corrupted reads likely result in a corrupted subchannel too, cdparanoia treats the subchannel as more trouble than it's worth (during verification). At least one other package (Exact Audio Copy for Win32) manages to use the subchannel to enhance the Table of Contents information. I don't know if this only works on MMC-2 drives that support returning Q with READ CD, but I think I'm going to revisit using the subchannel for extra TOC information. ----------------------------------------------------------------------- Does cdparanoia lose quality from the CD recording? Does it just re-record the analog signal played from the CDROM drive? No to both. Cdparanoia (and all other true CD digital audio extraction tools) reads the values off the CDROM in digital form. The data never comes anywhere near the soundcard, and does not pass through any conversion to analog first. ----------------------------------------------------------------------- Can cdparanoia detect pregaps? Can it remove the two second gaps between tracks Not yet. This feature is slated to appear in a release of alpha 10 (Paranoia IV). ----------------------------------------------------------------------- Why don't you implement CDDB? A GUI? Four million other features I want? Too many features spoil the broth. "Software is not perfect when there is nothing left to add, but rather when there is nothing extraneous left to take away." The goal of cdparanoia is perfect, rock-solid audio from every capable cdrom on every platform. As this goal has not yet been met, I'm uninterested in adding unrelated capability to the core engine. Several GUIs that incorporate cdparanoia already exist; I'm in the process of compiling a list (see the links page). Other software that implements new features by wrapping around cdpar anoia (like CDDB lookup) also exist. 'Cdparanoia' will not play to sound cards (you can always pipe the output to a WAV player), do MD5 signatures, read CD catalog or serial numbers (this *is* a feature I plan to add), search indexes, do rate reduction (use Sox, Ogg or a million others), or generally make use of the maximum speed available from a CDROM drive. If your CDROM drive is *not* prone to jitter and you don't have scratched discs to worry about, you might want to look at the original cdda2wav for features cdparanoia does not have. Keep in mind however that even the really good drives do occasionally stumble. I know of at least one cdparanoia user who insists on using full paranoia with his Plextor UltraPlex because it once botched a single sector from a rip; he'd already burned the track to several CD-Rs before noticing... ----------------------------------------------------------------------- The progress meter: What is that weird bargraph during ripping? It's a progress/status indicator. There's a completion bargraph, a number indicating the last sector number completely verified of the read currently happening, an overlap indicator, a gratuitous smilie, and a heartbeat indicator to show if the process is still alive, hung, or spinning. The bargraph also marks points during the read with characters to indicate where various 'paranoia' features were tripped into action. Different bargraph characters indicate different things occurred during that part of the read. The letters are heirarchical; for example if a trasport error occurs in the same sector as jitter, the bargraph will print 'e' instead of '-'. Legend of characters A hyphen indicates that two blocks overlapped properly, - but they were skewed (frame jitter). This case is completely corrected by Paranoia and is not a cause for concern. A plus indicates not only frame jitter, but an unreported, uncorrected loss of streaming in the middle + of an atomic read operation. That is, the drive lost its place while reading data, and restarted in some random incorrect location without alerting the kernel. This case is also corrected by Paranoia. An 'e' indicates that a transport level SCSI or ATAPI e error was caught and corrected. Paranoia will completely repair such an error without audible defects. An "X" indicates a scratch was caught and corrected. X Cdparanoia wil interpolate over any missing/corrupt samples. An asterisk indicates a scratch and jitter both * occurred in this general area of the read. Cdparanoia wil interpolate over any missing/corrupt samples. A ! indicates that a read error got through the stage one of error correction and was caught by stage two. Many '!' are a cause for concern; it means that the drive is making continuous silent errors that look ! identical on each re-read, a condition that can't always be detected. Although the presence of a '!' means the error was corrected, it also means that similar errors are probably passing by unnoticed. Upcoming releases of cdparanoia will address this issue. A V indicates a skip that could not be repaired or a V sector totally obliterated on the medium (hard read error). A 'V' marker generally results in some audible defect in the sample. The smilie is actually relevant. It makes different faces depending on the current errors it's correcting. Legend of smilies :-) Normal operation. No errors to report; if any jitter is present, it's small. :-| Normal operation, but average jitter is quite large. A rift was found in the middle of an atomically read block; in other words, the drive lost streaming in the :-P middle of a read and did not abort, alert the kernel , or restart in the proper location. The drive silently continued reading in so me random location. :-/ The read appears to be drifting; cdparanoia is shifting all of its reads to make up for it. Two matching vectors were found to disagree even after first stage verification; this is an indication that the drive is reliably dropping/adding bytes at consistent locations. Because the verification algorithm is partially 8-| based on rereading and comparing vectors, if two vectors read incorrectly but identically, cdparanoia may never detect the problem. This smilie indicates that such a situation *was* detected; other instances may be slipping through. Transport or drive error. This is normally not a cause for concern; cdparanoia can repair just about any error that :-0 it actually detects. For more information about these errors, run cdparanoia with the -v option. Any all all errors and a description will dump to stderr. :-( Cdparanoia detected a scratch. Cdparanoia gave up trying to repair a sector; it could not read consistent enough information from the drive to do ;-( so. At this point cdparanoia will make the best guess it has available and continue (a V appears in the bargraph at this point). This often results in an audible defect. Cdparanoia displays this smilie both when finished reading :^D a track and also if no error correction mechanism has been tripped so far reading a new track. ----------------------------------------------------------------------- How can I tell if my drive would be OK with regular cdda2wav? Easy. Run cdparanoia; if the progress meter never shows any characters but the little arrow going across the screen, the CDROM drive is probably one of the (currently) few drives that can read a pristine stream of data off an audio disc regardless of circumstances. This drive will work quite well with cdda2wav (or cdparanoia using the '-Z' option) A drive that results in a bargraph of all hyphens would *likely* work OK with cdda2wav, but it's less certain. Any other characters in the bargraph (colons, semicolons, pluses, Xs, etc..) indicate that a fixups had to be performed at that point during the read; that read would have failed or 'popped' using cdda2wav. ----------------------------------------------------------------------- What is the biggest value of SG_BIG_BUFF I can use? This is relevant only to 2.0 kernels and early 2.2 kernels. Modern Linux kernels no longer have a single static SG DMS pool. For 2.0, 65536 (64 kilobytes). Some motherboards can use 128kB DMA, but attempting to use 128kB DMA on a machine that can't do it will crash the machine. Cdparanoia will not use larger than 64kB requests. ----------------------------------------------------------------------- Why do the binary files from two reads differ when compared? The problem is the beginning point of the read. Cdparanoia enforces consistency from whatever the drive considers to be the starting point of the data, and the drive is returning a slightly different beginning point each time. The beginning point should not vary by much, and if this shift is accounted for when comparing the files, they should indeed turn out to be the same (aside from errors duly reported during the read; scratch correction or any reported skips will very likely also result in different files). ----------------------------------------------------------------------- Why do CDParanoia, CDDA2WAV et al. rip files off into WAV format (and other sample formats) but not CDDA format? WAV and AIFC are simply convenient formats that include enough header information such that multipurpose audio software can uniquely identify the form of the data in the sample. In raw form, mulaw, SND and CDDA look exactly alike to a program like xplay, and are very likely to blow your ears (and stereo) out when played! Header formats are more versatile and safer. By default, cdparanoia and cdda2wav write WAV files. That said, cdparanoia (and cdda2wav) will write raw, headerless formats if explicitly told to. Cdparanoia writes headerless, signed 16 bit, 44.1kHz stero files in little endian format (LSB first) when given the -r option, and the same in big endian (MSB) format when given -R. All files written by cdparanoia are a multiple of 2352 bytes long (minus the header, if any) as required by cd writer software. Cdparanoia and the Laser-Playback-Head-of-Omniscience logo are trademarks (tm) of Xiphophorus (xiph.org). This document copyright (C) 1994-1999 Xiphophorus. All rights reserved. Comments and questions are welcome.