Handbook¶

Install¶

We recommend using uv to install or run kalika. [1][2]

pip install uv
uvx kalika

Alternatively, if you like to install it account-wide on your system:

uv tool install kalika

Configure¶

Multiple or large instances of Heritrix processing many jobs require to increase certain system limits. Please define in /etc/sysctl.d/60-local.conf:

fs.inotify.max_user_instances = 10240
fs.inotify.max_user_watches = 20037690

Then, activate the new settings.

sysctl --system

Operate¶

Crawl¶

Run Heritrix.

docker run --detach --init --user root \
    --name heritrix-1 --rm --publish 8443:8443 \
    --env "USERNAME=admin" --env "PASSWORD=admin" --env "JAVA_OPTS=-Xmx4096M" \
    --volume /var/heritrix-1:/opt/heritrix/jobs docker.io/iipc/heritrix

Feed a list of URLs.

tmux new -s feed
export HERITRIX_URL="https://heritrix.example.org:8443/engine"
kalika heritrix add feed.txt

Drain WARC files into target directory.

tmux new -s drain
export HERITRIX_URL="https://heritrix.example.org:8443/engine"
kalika heritrix drain /media/archive

Replay¶

mkdir /media/replay
cd /media/replay
wb-manager init research-1
wb-manager add research-1 /media/archive/www.example.org.warc.gz
kalika serve --directory /media/replay

Tools¶

Display missing sites by comparing list of input URLs against files in the WARC output directory.

kalika urllist compare --url-list urls.txt --directory /media/archive

Create equal-sized chunks from URL file, named urls-00.txt, urls-01.txt, urls-02.txt, etc.

kalika urllist chunk urls.txt --chunk-size 500