End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

The End of Term (EOT) Web Archive Dataset presents a longitudinal dataset of the US federal web which includes publicly available .gov and .mil domains, created during the 2008, 2012, 2016, and 2020 presidential elections in the United States. Based on the End of Term Web Archive, this dataset presents 461TB of WARC data and accompanying derivative files in WAT, WET, and CDX format. A metadata sidecar file is also provided that contains content-based characterizations, including languages, character sets, format identifiers, and mime types. In addition to these derivative formats, CDX indexes in the ZipNum and Parquet formats that provide additional functionality to the dataset are included. The EOT dataset is freely available on the Amazon S3 platform as part of the Amazon Open Data Program.