Goals and Purposes of WARC

Allows for substantial information about the time of harvesting, the IP address of the harvesting machine, Internet Media Type (MIME type) and response code for the harvest transaction, the purpose of harvesting, etc.

Excellent for efficient bulk harvesting and efficient indexing for access by URL and date. The structured record headers can be extracted and stored separately for efficient indexing. WARC supports duplicate elimination and compression to reduce file sizes for storage, transmission, and indexing after harvesting.

Goals of the WARC file format include the following: 

•Ability to store both the payload content and control information from mainstream Internet application layer protocols, including HTTP, FTP, NNTP, and SMTP.

•Ability to store arbitrary metadata linked to other stored data (e.g., subject classifier, discovered language, encoding)

•Support for data compression and maintenance of data record integrity.

•Ability to store all control information from the harvesting protocol (e.g., request headers), not just response information.

•Ability to store the results of data transformations linked to other stored data.

•Ability to store a duplicate detection event linked to other stored data.

•Amenable to efficient processing.

•Sufficiently different from the legacy ARC format files that software tools can unambiguously detect and correctly process both WARC and ARC records.

•Ability to store globally unique record identifiers.

•Support for deterministic handling of long records (e.g., truncation, segmentation).