
주어진 URL에 대한 Wayback Machine 전체 아카이브를 다운로드합니다.
Waybackpack은 주어진 URL에 대한 Wayback Machine의 전체 아카이브를 다운로드할 수 있는 커맨드라인 도구입니다.
예를 들어, 미국 노동부 홈페이지의 1996년(해당 사이트가 처음 아카이브된 해)까지의 모든 사본을 다운로드하려면 다음을 실행하면 됩니다:
waybackpack http://www.dol.gov/ -d ~/Downloads/dol-wayback --to-date 1996
결과:
~/Downloads/dol-wayback/
├── 19961102145216
│ └── www.dol.gov
│ └── index.html
├── 19961103063843
│ └── www.dol.gov
│ └── index.html
├── 19961222171647
│ └── www.dol.gov
│ └── index.html
└── 19961223193614
└── www.dol.gov
└── index.html
또는 아카이브된 모든 스냅샷의 URL만 출력하려면:
waybackpack http://www.dol.gov/ --list
pip install waybackpack
usage: waybackpack [-h] [--version] (-d DIR | --list) [--raw] [--root ROOT]
[--from-date FROM_DATE] [--to-date TO_DATE]
[--user-agent USER_AGENT] [--follow-redirects]
[--uniques-only] [--collapse COLLAPSE] [--ignore-errors]
[--max-retries MAX_RETRIES] [--no-clobber] [--quiet]
[--progress] [--delay DELAY] [--delay-retry DELAY_RETRY]
url
positional arguments:
url The URL of the resource you want to download.
options:
-h, --help show this help message and exit
--version show program's version number and exit
-d DIR, --dir DIR Directory to save the files. Will create this
directory if it doesn't already exist.
--list Instead of downloading the files, only print the list
of snapshots.
--raw Fetch file in its original state, without any
processing by the Wayback Machine or waybackpack.
--root ROOT The root URL from which to serve snapshotted
resources. Default: 'https://web.archive.org'
--from-date FROM_DATE
Timestamp-string indicating the earliest snapshot to
download. Should take the format YYYYMMDDhhss, though
you can omit as many of the trailing digits as you
like. E.g., '201501' is valid.
--to-date TO_DATE Timestamp-string indicating the latest snapshot to
download. Should take the format YYYYMMDDhhss, though
you can omit as many of the trailing digits as you
like. E.g., '201604' is valid.
--user-agent USER_AGENT
The User-Agent header to send along with your requests
to the Wayback Machine. If possible, please include
the phrase 'waybackpack' and your email address. That
way, if you're battering their servers, they know who
to contact. Default: 'waybackpack'.
--follow-redirects Follow redirects.
--uniques-only Download only the first version of duplicate files.
--collapse COLLAPSE An archive.org `collapse` parameter. E.g., --collapse
timestamp:6 collapses results to at most one per
month. See here: https://github.com/internetarchive/wa
yback/blob/master/wayback-cdx-
server/README.md#collapsing
--ignore-errors Don't crash on non-HTTP errors e.g., the requests
library's ChunkedEncodingError. Instead, log error and
continue. See here:
https://github.com/jsvine/waybackpack/issues/19
--max-retries MAX_RETRIES
How many times to try accessing content with 4XX or
5XX status code before skipping?
--no-clobber If a file is already present (and >0 filesize), don't
download it again.
--quiet Don't log progress to stderr.
--progress Print a progress bar. Mutes the default logging.
Requires `tqdm` to be installed.
--delay DELAY Sleep X seconds between each fetch.
--delay-retry DELAY_RETRY
Sleep X seconds between each post-error retry.
Waybackpack은 순수 Python으로 작성되었으며, requests에만 의존하고 Python이 동작하는 모든 환경에서 작동합니다. Python 3.3+가 필요합니다.
버그를 발견하고, 오타를 수정하며, 유용한 기능을 제안해 주신 다음 사용자분들께 감사드립니다: