Difference between revisions of "LOCKSS Software"
m (→Repair Crawler) |
|||
| Line 181: | Line 181: | ||
Sets this to true in properties and repair will be done from publisher only | Sets this to true in properties and repair will be done from publisher only | ||
org.lockss.crawler.num_retries_from_caches [5] | org.lockss.crawler.num_retries_from_caches [5] | ||
| − | + | Sets this in properties to limit the number of caches it will try in repair mode 1,3 and 4.</nowiki> | |
The repair crawler must be called with a list of specific URLs. The crawler does not discover new content by following links. | The repair crawler must be called with a list of specific URLs. The crawler does not discover new content by following links. | ||
Revision as of 15:08, 18 June 2013
LOCKSS Software
Source
- Source : http://sourceforge.net/scm/?type=cvs&group_id=47774 (SourceForge)
- ViewVC : http://lockss.cvs.sourceforge.net/viewvc/lockss/ (view it online)
- Release Notes : http://www.lockss.org/support/use-a-lockss-box/daemon-release-notes/
File System
Primary file system distribution of LOCKSS content.
- /etc/lockss
- /home/lockss
- /usr/share/lockss
- /var/log/lockss
In the file /etc/lockss/config.dat, the LOCKSS_DISK_PATHS variable enumerates the available disks.
- /cache0/gamma
- /cache1/gamma
- /cache2/gamma
- /cache3/gamma
File System Deep Dive
Access URL http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/tif/000404.tif
Cache storage /cache0/gamma/cache/m/bpldb.bplonline.org/http/adpn/load/Cartography/000400-000599/000404/tif/000404.tif/
Where are the bytes of this tif file?
/cache0/gamma/cache/m/bpldb.bplonline.org/http/adpn/load/Cartography/000400-000599/000404/tif/000404.tif/#content/current
current is a raw byte file. If the content is binary, the file is binary. If the content is text, the file is text. In order to understand the type of file current is you should examine current.props.
/cache0/gamma/cache/m/bpldb.bplonline.org/http/adpn/load/Cartography/000400-000599/000404/tif/000404.tif/#content/current.props describes the bytes (HTTP headers, LOCKSS specific headers)
/cache0/gamma/cache/m
#au_id_file
#au_state.xml
#id_agreement.xml
#no_au_peers
#node_props
#nodestate.xml
bpldb.bplonline.org/
http/
#node_props
adpn/
#node_props
load/
#node_props
Cartography/
#node_props
000400-000599/
#agreement
#node_props
#content/
manifest.html/
000404/
#agreement
#node_props
#content/
tif/
#agreement
#node_props
#content/
000404.tif/
#agreement
#node_props
#content/
current
current.props
Revisions
Each revision is stored with a full copy of the bytes. current and current.props are renamed to 1 and 1.props. Each revision can always be retrieved.
/cache0/gamma/cache/m/bpldb.bplonline.org/http/adpn/load/Cartography/000400-000599/000404/tif/000404.tif/
#content/
1
1.props
2
2.props
current
current.props
Starting and Stopping the LOCKSS daemon
- /etc/init.d/lockss start
- /etc/init.d/lockss stop
- /etc/init.d/lockss restart
Log Rotate
Logs are in /var/log/lockss
logrotate /etc/logrotate.d/lockss
Configuration Files
Primary Configuration files
- http://props.lockss.org:8001/adpn/lockss.xml
- /etc/lockss/config.dat
- /home/lockss/local.txt
- /cache0/gamma/config/expert_config.txt
- /cache0/gamma/config/au.txt
Additional Configuration files
- /cache0/gamma/config/ui_ip_access.txt
- /cache0/gamma/config/proxy_ip_access.txt
- /cache0/gamma/config/content_servers_config.txt
Configuration parameters are listed
- http://www.lockss.org/lockssdoc/gamma/daemon/paramdoc.html
- http://www.lockss.org/lockssdoc/gamma/daemon/paramdoc.txt
Debug options from Logger.java
- info -- default generally,
- debug -- debugging messages, sparse
- debug1 -- debugging messages
- debug2 -- Detailed debugging that would not produce a ridiculous amount of output if it were enabled system-wide.
- debug3 -- Debugging messages that produce more output than would be reasonable if this level were enabled system-wide. (e.g. messages in inner loops, or per-file, per-hash step, etc.)
Examples of use of log level in Expert Config
org.lockss.log.BaseCrawler.level = info org.lockss.log.CrawlerImpl.level = info org.lockss.log.BlockTally.level = info org.lockss.log.V3PollerStatus.level = debug org.lockss.log.PlatformInfo.level = debug2
Title List
The default title lists for available AUs is provided by LOCKSS in http://props.lockss.org:8001/adpn/lockss.xml. The Add/Remove title lists in the Web admin are populated from titledb.xml.
<property name="titleDbs"> <list> <value>http://props.lockss.org:8001/adpn/titledb/titledb.xml</value> </list> </property>
It is possible to overwrite the default configuration using a local parameter org.lockss.titleDbs.
org.lockss.titleDbs = http://bpldb.bplonline.org/etc/adpn/titledb-local.xml
Web Crawler
There are two distinct crawlers in LOCKSS: the new content crawler and repair crawler. The new content crawler is the primary method of inserting data in the cache and the crawler will only start at the publisher's content staging area as defined in the AU start URL settings. The repair crawler, depending on configuration parameters, can repair from the publisher and/or peer nodes. The Web crawler supports HTTP and HTTPS URL protocols.
Plugins
Plugins can be retrieved from the LOCKSS node or Source Forge http://lockss.cvs.sourceforge.net/viewvc/lockss/lockss-daemon/plugins/src/
- Plugin Tool Download (0.12.1 latest from SourceForge)
- LOCKSS Plugin Tool Tutorial (Way Back Machine IA)
Example plugin : BirminghamPublicLibraryBasePlugin.xml
New Content Crawler
The new content crawler will only retrieve data from the publisher's content staging area. It will follow links and discover new content.
Repair Crawler
Repair crawler modes
- repair from other caches only
- repair from publisher only
- repair from other caches first. If repair fails in trying a certain number of caches, it will try repair from the publisher
- repair from publisher first. If repair fails, it will try repair from other caches
Parameters to set modes,
org.lockss.crawler.fetch_from_other_caches_only [false] Sets this to true in properties and repair will be done from other caches only org.lockss.crawler.fetch_from_publisher_only [true] Sets this to true in properties and repair will be done from publisher only org.lockss.crawler.num_retries_from_caches [5] Sets this in properties to limit the number of caches it will try in repair mode 1,3 and 4.
The repair crawler must be called with a list of specific URLs. The crawler does not discover new content by following links.
Content Crawler and Revisions
Depending AU and plugin definitions, the repository manager will initiate a new content crawl for an existing AU. When examining previously crawled URLs, the LOCKSS crawler does not request a full copy of the remote data. The LOCKSS crawler retrieves the last modified date of the cached content in stored metadata (current.props) and sends that date as an HTTP header (If-Modified-Since) to the remote server. The remote server checks file last modified date against the LOCKSS request HTTP header data and responds with 304 (not-modified) when the file has not changed. In the event the remote server determines the file last modified date is newer than LOCKSS request HTTP header date, the remote server sends the content.
According to RepositoryNodeImpl.java (see sealNewVersion()), the LOCKSS repository manager only saves crawler content in the cache as a revision if the files are not identical. If files are identical but last modified time is different, the repository manager renames the existing current.props to current.props-[LAST MODIFIED DATE] and the most recent file metadata is saved into current.props. current isn't touched.
Manipulating the Cache
Exporting Content
The LOCKSS repository manager provides a simple HTTP-based download of any content type on the LOCKSS server by linking, copying or moving the data to /cache0/gamma/tmp/export/ and visiting Web admin http://bpl-adpnet.jclc.org:8081/ExportContent
/cache0/gamma/tmp/export/ is emptied on daemon restart, unless file ownership is changed from LOCKSS.
Deleting Stale AUs
Repositories marked as 'Deleted' in the RepositoryTable can be removed from the cache by deleting the appropriate directory (i.e. /cache0/gamma/cache/f). AUs not marked as deleted should be removed through the Web UI first or, untested, by removing the appropriates lines from /cache0/gamma/config/au.txt. Then remove the directory in the cache.
Restart the daemon.
Moving AUs, Same Disk
The daemon fills in missing directories when adding AUs so renaming in a directory is not strictly necessary. Nevertheless, cache directories are incremented 'a-z' then 'a[a-z]', 'b[a-z]' etc. An AU is not referenced by a specific cache directory location. An AU is referenced by indicating the disk repository in au.txt (ie. local\:/cache0/gamma NOT local\:/cache0/gamma/f). To move an AU to a new location on the same disk, simply rename the directory.
Consider the following scenario, AUs exist at 'a', 'b', 'c', 'd', 'e'. After deleting AUs 'c' and 'd', cache/gamma looks like 'a', 'b', 'e'. I can rename directory 'e' to 'c' and the next AU added will be appropriately named 'd'.
Restart the daemon.
n.b. Wait until active crawls are completed before renaming directories, otherwise crawler will use initial directories
Moving AUs, Different Disk
Cache directories are incremented "a-z" then "a[a-z]", "b[a-z]" etc. first identify the next directory name on the new disk.
Locate and modify the appropriate lines defining the AU in /cache0/gamma/config/au.txt (e.g. repository=local\:/cache1/gamma changes to local\:/cache2/gamma). Move the directory from the original to the new disk (e.g. mv /cache1/gamma/cache/g /cache2/gamma/cache/gj)
Restart the daemon.
Copying AUs, Different Nodes
The repository manager of a node will not initiate a new content crawl to a peer node, and new content crawls are generally recognized as the only way to populate a node's cache. It is possible to copy a file system from on node and transfer it to another. Once the node is populated with data and the repository manager is coerced into believing a new content crawl was completed successfully, the repository manager will engage in normal peer polling for the archival unit. Modification or addition of data to the archival unit still requires the use of the publisher's content staging area with the original URL structure.
Understand File System Organization
File system organization in the cache is system dependent only to a depth of 4 from the root. After that point, the file system organization is dependent on the original access URL for the archival unit data.
It is possible to populate a new node using a copy of the file system of a peer node. The peer node should have indicated a high-reliability factor of the cached data (100% agreement in a recent poll would be good). (I am shortcutting some steps under the assumption that the archival unit is in the add titles list of the LOCKSS Web UI, otherwise one would have to manually edit the au.txt file and create directories in the cache.)
/cache0/gamma/cache/m/bpldb.bplonline.org/http/adpn/load/Cartography/000400-000599/000404/tif/000404.tif/
When the archival unit is added to the node through the Web UI, the repository manager creates the next available directory based on the volume selected. (The base directory for the archival unit truncates at /cache0/gamma/cache/m/, the extra path in the example is to show URL structure dependence). The repository manager will then initiate a new content crawl to the start URL defined in the AU. Since the publisher has vacated the content staging area, the crawl will result in a 404 or other HTTP error. The changes to au.txt and creation of the /m/ directory in the file system are not rolled back.
Pack the AU
A tarred (compressed) package of an archival unit should be made on the peer node with content. For this example, the peer node had the archival unit in the file system at /cache2/gamma/cache/bj/.
tar -zpcvf /cache0/gamma/tmp/export/au_to_transfer.tar.gz /cache2/gamma/cache/bj/
There are a number of methods of exporting the content from the LOCKSS node, including FTP, but for this example I am utilizing the built-in HTTP export. After this command is executed, there will be a link called "au_to_transfer.tar.gz" in the ExportContent page of the LOCKSS Web UI.
Unpack the AU
The present working directory at this point should be /cache0/gamma/cache/. The appropriate command
tar -xvzf /path/to/au_to_transfer.tar.gz -C m --strip-components=4 --exclude \#agreement --exclude \#no_au_peers --exclude \#id_agreement.xml --exclude \#node_props --exclude \#au_id_file
Finish Up
The #node_props files will be generated at the LOCKSS daemon restart, which should be done at this time. By preserving the files #au_state.xml and #nodestate.xml, the repository manager thinks that a new content crawl has already been completed. A manually initiated V3 Poll on the archival unit should return appropriate URL agreement levels. The new LOCKSS node should now engage in normal content polling with the other peers.
Voting and Polling
Hashed UrlSets
AU released by LOCKSS with current content staging
# Block hashes from bpl-adpnet.jclc.org, 12:00:19 11/07/12 # AU: Birmingham Public Library Cartography Collection: Maps (000400-000599) # Hash algorithm: SHA-1 # Encoding: Base64 IpncVSUBDZaqglSsjkOp49OS4KE= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/ M7+kc1l4Nl/Jwp3Hp0XRgK8dK94= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/mrc lMeB8SlN+CGI7+1LxRhE+btzGmo= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/mrc/000404.mrc 266QzyI+r7sZVcRZGId2/roOLNI= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/tif Tv2dnIhIPXEllk95fqAeA8uB67s= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/tif/000404.tif 6lm8iDE+/gu5ZWcJxTFjU5PP6OI= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/txt b5Ta9uvYyS0GSZ7srapk2YZGWJM= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/txt/000404.csv iuzvdBZG0oft3zxao5Sn5r7m+dA= http://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/txt/000404.txt # end
Manually created AU with limited copy of current content
# Block hashes from bpl-adpnet.jclc.org, 11:41:58 11/07/12 # AU: Birmingham Public Library Base Plugin, Base URL http://bpldb.bplonline.org/adpn/load/, Group test, Collection 000404-000404 # Hash algorithm: SHA-1 # Encoding: Base64 GZzfEqeNHw8WdjuZbX1g/VrJyaI= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/ JkkpA1r7E548EOiiu0KeBuQYfts= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/mrc lMeB8SlN+CGI7+1LxRhE+btzGmo= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/mrc/000404.mrc uVD1FzG1WN43ROniD/VfQb/P6pU= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/tif Tv2dnIhIPXEllk95fqAeA8uB67s= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/tif/000404.tif 0bG54sCMTETu5qKnBA3RXO80v/0= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/txt b5Ta9uvYyS0GSZ7srapk2YZGWJM= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/txt/000404.csv iuzvdBZG0oft3zxao5Sn5r7m+dA= http://bpldb.bplonline.org/adpn/load/test/000404-000404/000404/txt/000404.txt # end
VoteBlock
url=http\://bpldb.bplonline.org/adpn/load/Cartography/000400-000599/000404/tif/000404.tif vn=I1dlZCBOb3YgMDcgMDM6NTg6NTYgQ1NUIDIwMTIKdW89MApmbz0wCnVsPTQ0NzkxMTEwCm5oPWw1S2NNcnFOdS9McVlobWE3RFNQcm13ZEhab1w9CmVycj1mYWxzZQpmbD00NDc5MTExMApwaD1UdjJkbkloSVBYRWxsazk1ZnFBZUE4dUI2N3NcPQo\= vt=0 (#Wed Nov 07 03:58:56 CST 2012
VoteBlock Versions (vn) decoded from Base64. Notice the ph (Plain Hash) for item 000404.tif matches both instances in the Hashed UrlSet.
#Wed Nov 07 03:58:56 CST 2012 uo=0 fo=0 ul=44791110 nh=l5KcMrqNu/LqYhma7DSPrmwdHZo\= err=false fl=44791110 ph=Tv2dnIhIPXEllk95fqAeA8uB67s\=
BlockTally
BlockTally.java enumerates poll results including TOO_CLOSE and NOQUORUM.
TOO_CLOSE
Configuration parameter
org.lockss.poll.v3.voteMargin [75]
Need to investigate this further, particularly conflicts with quorum.
Does this mean a poll of 5 voters with 3 votes in agreement is invalid because (3/5)*100 = 60?
voteMargin input parameter is from configuration or default (75).
boolean isWithinMargin(int voteMargin) {
int numAgree = agreeVoters.size();
int numDisagree = disagreeVoters.size();
double numVotes = numVotes();
double actualMargin;
if (numAgree > numDisagree) {
actualMargin = (double) numAgree / numVotes;
} else {
actualMargin = (double) numDisagree / numVotes;
}
if (actualMargin * 100 < voteMargin) {
return false;
}
return true;
}
NOQUORUM
Configuration parameter
org.lockss.poll.v3.quorum [5]
ADPNet V3Poll quorum configuration is 3.
Polling minimum is 3, not 3 + caller.
11:15:56.170: Debug: 10-V3Poller: [InvitationCallback] Enough peers are participating (3+0)