We believe that it is important for people to be able to make good quality local copies of Erowid (or any website). Erowid is dedicated to making the site as useful to as many people as possible and we believe that an important function of library sites is that they be friendly to archivers and offline readers. However, the issue is complicated by practical performance issues, bandwidth constraints, and technical hurdles that make this somewhat more difficult than it may first appear.

A huge load is placed on the Erowid server by spiders and robots that attempt to download the entire site. We have been working to produce an easily downloaded version of the site as well as mirror servers that can be more easily spidered and archived. Many archiver robots do a very poor job with large, complicated websites especially those that use javascript navigation.

In 2004, with the help of JL, we were able to launch our first mirror of the site on a server in Germany (de1.erowid.org). In June 2005, with the help of Stu, we now have a system in place for making regular mirrors of the site and downloadable tar files that can be used to put Erowid on CD or on a local harddrive. This is based on the work we've been doing on Sperowider, an archiver robot designed to work on the Erowid site.

With the world-wide reach of Erowid.org, it is not uncommon for dozens of spiders to be running concurrently and one poorly behaved spider can slow down access for every visitor to the site. If you use spidering software or an archiver robot on Erowid, you will be expected to keep very close watch on it as it runs. If it hits the server for more than 10 hits per second or downloads more than 1,000 pages in a day without following our guidelines, your access to the site may be blocked.

Happily, there are several ways you can now archive Erowid in a community-friendly way:

1. Become a member and get Erowid on CD. Make a qualifying donation to Erowid and get a copy of Erowid on CD sent to you in the mail. [See details]
2. Download a recent archive from one of our download locations, rather than attempting to spider the site on your own. [See archive locations]
3. Spider a mirror site instead of the main server. The flat-file archives of the site that are used for the mirrors make spidering much easier and less prone to out-of-contol robots becoming unintentional Denial of Service attacks. Please direct your robot at one of the available Erowid Mirrors.
4. Request permission to spider the live site. Our primary requirements for spidering of the live site:
   1. Run your spider only during off-peak hours (between midnight and 5am PST)
   2. Limit your robot to 10 file hits per second during off-peak hours and 2 file hits per second during peak hours.
   3. Robots must properly identify themselves. Archiving and spidering robots that do not identify themselves or that falsify their agent name will result in your IP(s) being blocked. Please set up your spider to do very short runs (less than 100 pages) and test it while you monitor the robot before simply launching it against our site and walking away.
   Please contact sage@erowid.org if you'd like to have your spidering approved.
5. Let us know what sub-sections you'd like to download. We can try to provide customized slices of the site for you if you let us know what you'd like to see. If you are a current member or donor, obviously we'll be more excited about spending our time trying to help you with customized archives. ;]

• de1.erowid.org/erowid_archives/ : Hosted by JL

• de1.erowid.org : Hosted by JL