http://www.adpn.org/w/api.php?action=feedcontributions&user=CJohnson&feedformat=atomAdpnwiki - User contributions [en]2024-03-28T12:50:41ZUser contributionsMediaWiki 1.35.3http://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2059HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:57:55Z<p>CJohnson: /* 8. Initiate a request to crawl the newly-added AU */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* '''You'll need: the Peer Code for your [[Preservation Node]].''' (See above.) <br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
The '''Expert Config''' interface provides a simple text editing box with a series of key-value pairs, one on each line (in the format `key=value`):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png]]<br />
<br />
The setting that you want to change is '''org.lockss.titleDbs'''. Normally, it will be set to point to the published <code>lockss.xml</code> file on the [[props server]] (<code>http://configuration.adpn.org/lockss.xml</code>).<br />
<br />
You want to change it to a new URL that dynamically includes AUs accepted for a [[test crawl]], but not yet published to the entire network. The URL you need will use the [[Peer Code]] that you noted above. For example, if you are performing a [[test crawl]] on the [[Preservation Node]] with the code <code>FOO</code>, you would use the URL and setting:<br />
<br />
: org.lockss.titleDbs=http://configuration.adpn.org/titlelist/index?peer=FOO&stype=1&ext=.xml<br />
<br />
You should also preserve the old value for this setting, so that you can easily revert back to the old setting when you are done performing and confirming the [[test crawl]]. To do this, just edit the old line to change the name of the key to an altered name, such as <code>org.lockss.titleDbs.0</code>. Then insert the new line above the old setting. For example, here is what we would use at the Alabama Department of Archives and History (Peer Code <code>ADAH</code>):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png]]<br />
<br />
Mash the '''Update''' button to save your changed settings.<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
<br />
== 8. Initiate a request to crawl the newly-added AU ==<br />
<br />
LOCKSS will often begin crawling a new AU relatively soon after it has been added to the daemon's list of AUs; but you can and should use the ''''Start Crawl''' button in the LOCKSS daemon '''Debug Panel''' to help the process speed along. (The label for this button is slightly misleading -- it does not actually force LOCKSS to start the HTTP crawl of the selected AU immediately. But it does boost the crawl right up to the top of the priority queue for pending crawls, so it should guarantee that it starts relatively soon.)<br />
<br />
To do this, go to your LOCKSS daemon's administrative interface and select the '''Debug Panel''' from the navigation links on the left. Then, find the drop-down list located just above the '''Start V3 Poll''' and '''Start Crawl''' buttons. Pull down the drop-down and scroll through the list of AUs to find the new one which you have just added:<br />
<br />
[[File:Screenshot-adpnadah-debug-panel-au-dropdown.jpg]]<br />
<br />
Then, once the AU is selected, mash the '''Start Crawl''' button:<br />
<br />
[[File:Screenshot-adpnadah-start-crawl-au-selected.png]]<br />
<br />
Then you can verify that the crawl has been put into the queue and set to '''Pending''' by going to '''Daemon Status''' > '''Active Crawls'''<br />
<br />
[[File:Screenshot-adpnadah-daemon-status-confirm-crawl.png]]<br />
<br />
If it's there, then you can get some coffee, do something else for the next several hours, and wait for the LOCKSS daemon to initiate and complete the test crawl.<br />
<br />
== 9. After a while, confirm that the AU has been successfully crawled ==<br />
<br />
== 10. In Expert Config, reset your titlesDb URL to its original value ==<br />
<br />
When you have '''COMPLETED''' the [[test crawl]] and '''CONFIRMED''' its success, you will usually want to reset your '''Expert Config''' settings so that your LOCKSS daemon will pull AUs from the network-standard published <code>lockss.xml</code> file, instead of from the test feed. To do this, reset the value of your <code>org.lockss.titleDbs</code> setting to the original URL:<br />
<br />
[[File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-daemon-status-confirm-crawl.png&diff=2058File:Screenshot-adpnadah-daemon-status-confirm-crawl.png2022-07-08T16:56:35Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-start-crawl-au-selected.png&diff=2057File:Screenshot-adpnadah-start-crawl-au-selected.png2022-07-08T16:54:55Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-debug-panel-au-dropdown.jpg&diff=2056File:Screenshot-adpnadah-debug-panel-au-dropdown.jpg2022-07-08T16:53:30Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2055HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:47:16Z<p>CJohnson: /* 10. In Expert Config, reset your titlesDb URL to its original value */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* '''You'll need: the Peer Code for your [[Preservation Node]].''' (See above.) <br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
The '''Expert Config''' interface provides a simple text editing box with a series of key-value pairs, one on each line (in the format `key=value`):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png]]<br />
<br />
The setting that you want to change is '''org.lockss.titleDbs'''. Normally, it will be set to point to the published <code>lockss.xml</code> file on the [[props server]] (<code>http://configuration.adpn.org/lockss.xml</code>).<br />
<br />
You want to change it to a new URL that dynamically includes AUs accepted for a [[test crawl]], but not yet published to the entire network. The URL you need will use the [[Peer Code]] that you noted above. For example, if you are performing a [[test crawl]] on the [[Preservation Node]] with the code <code>FOO</code>, you would use the URL and setting:<br />
<br />
: org.lockss.titleDbs=http://configuration.adpn.org/titlelist/index?peer=FOO&stype=1&ext=.xml<br />
<br />
You should also preserve the old value for this setting, so that you can easily revert back to the old setting when you are done performing and confirming the [[test crawl]]. To do this, just edit the old line to change the name of the key to an altered name, such as <code>org.lockss.titleDbs.0</code>. Then insert the new line above the old setting. For example, here is what we would use at the Alabama Department of Archives and History (Peer Code <code>ADAH</code>):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png]]<br />
<br />
Mash the '''Update''' button to save your changed settings.<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
<br />
== 8. Initiate a request to crawl the newly-added AU ==<br />
<br />
== 9. After a while, confirm that the AU has been successfully crawled ==<br />
<br />
== 10. In Expert Config, reset your titlesDb URL to its original value ==<br />
<br />
When you have '''COMPLETED''' the [[test crawl]] and '''CONFIRMED''' its success, you will usually want to reset your '''Expert Config''' settings so that your LOCKSS daemon will pull AUs from the network-standard published <code>lockss.xml</code> file, instead of from the test feed. To do this, reset the value of your <code>org.lockss.titleDbs</code> setting to the original URL:<br />
<br />
[[File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2054HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:46:58Z<p>CJohnson: /* 9. In Expert Config, reset your titlesDb URL to its original value */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* '''You'll need: the Peer Code for your [[Preservation Node]].''' (See above.) <br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
The '''Expert Config''' interface provides a simple text editing box with a series of key-value pairs, one on each line (in the format `key=value`):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png]]<br />
<br />
The setting that you want to change is '''org.lockss.titleDbs'''. Normally, it will be set to point to the published <code>lockss.xml</code> file on the [[props server]] (<code>http://configuration.adpn.org/lockss.xml</code>).<br />
<br />
You want to change it to a new URL that dynamically includes AUs accepted for a [[test crawl]], but not yet published to the entire network. The URL you need will use the [[Peer Code]] that you noted above. For example, if you are performing a [[test crawl]] on the [[Preservation Node]] with the code <code>FOO</code>, you would use the URL and setting:<br />
<br />
: org.lockss.titleDbs=http://configuration.adpn.org/titlelist/index?peer=FOO&stype=1&ext=.xml<br />
<br />
You should also preserve the old value for this setting, so that you can easily revert back to the old setting when you are done performing and confirming the [[test crawl]]. To do this, just edit the old line to change the name of the key to an altered name, such as <code>org.lockss.titleDbs.0</code>. Then insert the new line above the old setting. For example, here is what we would use at the Alabama Department of Archives and History (Peer Code <code>ADAH</code>):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png]]<br />
<br />
Mash the '''Update''' button to save your changed settings.<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
<br />
== 8. Initiate a request to crawl the newly-added AU ==<br />
<br />
== 9. After a while, confirm that the AU has been successfully crawled ==<br />
<br />
== 10. In Expert Config, reset your titlesDb URL to its original value ==<br />
<br />
When you have '''COMPLETED'' the [[test crawl]] and '''CONFIRMED''' its success, you will usually want to reset your '''Expert Config''' settings so that your LOCKSS daemon will pull AUs from the network-standard published <code>lockss.xml</code> file, instead of from the test feed. To do this, reset the value of your <code>org.lockss.titleDbs</code> setting to the original URL:<br />
<br />
[[File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2053HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:46:51Z<p>CJohnson: /* 7. Select the new AU and mash "Add Selected AUs" */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* '''You'll need: the Peer Code for your [[Preservation Node]].''' (See above.) <br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
The '''Expert Config''' interface provides a simple text editing box with a series of key-value pairs, one on each line (in the format `key=value`):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png]]<br />
<br />
The setting that you want to change is '''org.lockss.titleDbs'''. Normally, it will be set to point to the published <code>lockss.xml</code> file on the [[props server]] (<code>http://configuration.adpn.org/lockss.xml</code>).<br />
<br />
You want to change it to a new URL that dynamically includes AUs accepted for a [[test crawl]], but not yet published to the entire network. The URL you need will use the [[Peer Code]] that you noted above. For example, if you are performing a [[test crawl]] on the [[Preservation Node]] with the code <code>FOO</code>, you would use the URL and setting:<br />
<br />
: org.lockss.titleDbs=http://configuration.adpn.org/titlelist/index?peer=FOO&stype=1&ext=.xml<br />
<br />
You should also preserve the old value for this setting, so that you can easily revert back to the old setting when you are done performing and confirming the [[test crawl]]. To do this, just edit the old line to change the name of the key to an altered name, such as <code>org.lockss.titleDbs.0</code>. Then insert the new line above the old setting. For example, here is what we would use at the Alabama Department of Archives and History (Peer Code <code>ADAH</code>):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png]]<br />
<br />
Mash the '''Update''' button to save your changed settings.<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
<br />
== 8. Initiate a request to crawl the newly-added AU ==<br />
<br />
== 9. After a while, confirm that the AU has been successfully crawled ==<br />
<br />
== 9. In Expert Config, reset your titlesDb URL to its original value ==<br />
<br />
When you have '''COMPLETED'' the [[test crawl]] and '''CONFIRMED''' its success, you will usually want to reset your '''Expert Config''' settings so that your LOCKSS daemon will pull AUs from the network-standard published <code>lockss.xml</code> file, instead of from the test feed. To do this, reset the value of your <code>org.lockss.titleDbs</code> setting to the original URL:<br />
<br />
[[File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2052HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:45:52Z<p>CJohnson: /* 8. In Expert Config, reset your titlesDb URL to its original value */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* '''You'll need: the Peer Code for your [[Preservation Node]].''' (See above.) <br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
The '''Expert Config''' interface provides a simple text editing box with a series of key-value pairs, one on each line (in the format `key=value`):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png]]<br />
<br />
The setting that you want to change is '''org.lockss.titleDbs'''. Normally, it will be set to point to the published <code>lockss.xml</code> file on the [[props server]] (<code>http://configuration.adpn.org/lockss.xml</code>).<br />
<br />
You want to change it to a new URL that dynamically includes AUs accepted for a [[test crawl]], but not yet published to the entire network. The URL you need will use the [[Peer Code]] that you noted above. For example, if you are performing a [[test crawl]] on the [[Preservation Node]] with the code <code>FOO</code>, you would use the URL and setting:<br />
<br />
: org.lockss.titleDbs=http://configuration.adpn.org/titlelist/index?peer=FOO&stype=1&ext=.xml<br />
<br />
You should also preserve the old value for this setting, so that you can easily revert back to the old setting when you are done performing and confirming the [[test crawl]]. To do this, just edit the old line to change the name of the key to an altered name, such as <code>org.lockss.titleDbs.0</code>. Then insert the new line above the old setting. For example, here is what we would use at the Alabama Department of Archives and History (Peer Code <code>ADAH</code>):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png]]<br />
<br />
Mash the '''Update''' button to save your changed settings.<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
== 9. In Expert Config, reset your titlesDb URL to its original value ==<br />
<br />
When you have '''COMPLETED'' the [[test crawl]] and '''CONFIRMED''' its success, you will usually want to reset your '''Expert Config''' settings so that your LOCKSS daemon will pull AUs from the network-standard published <code>lockss.xml</code> file, instead of from the test feed. To do this, reset the value of your <code>org.lockss.titleDbs</code> setting to the original URL:<br />
<br />
[[File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png&diff=2051File:Screenshot-adpnadah-20220708-1142-expert-config-edited-published-url.png2022-07-08T16:43:55Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2050HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:41:06Z<p>CJohnson: /* 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* '''You'll need: the Peer Code for your [[Preservation Node]].''' (See above.) <br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
The '''Expert Config''' interface provides a simple text editing box with a series of key-value pairs, one on each line (in the format `key=value`):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png]]<br />
<br />
The setting that you want to change is '''org.lockss.titleDbs'''. Normally, it will be set to point to the published <code>lockss.xml</code> file on the [[props server]] (<code>http://configuration.adpn.org/lockss.xml</code>).<br />
<br />
You want to change it to a new URL that dynamically includes AUs accepted for a [[test crawl]], but not yet published to the entire network. The URL you need will use the [[Peer Code]] that you noted above. For example, if you are performing a [[test crawl]] on the [[Preservation Node]] with the code <code>FOO</code>, you would use the URL and setting:<br />
<br />
: org.lockss.titleDbs=http://configuration.adpn.org/titlelist/index?peer=FOO&stype=1&ext=.xml<br />
<br />
You should also preserve the old value for this setting, so that you can easily revert back to the old setting when you are done performing and confirming the [[test crawl]]. To do this, just edit the old line to change the name of the key to an altered name, such as <code>org.lockss.titleDbs.0</code>. Then insert the new line above the old setting. For example, here is what we would use at the Alabama Department of Archives and History (Peer Code <code>ADAH</code>):<br />
<br />
[[File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png]]<br />
<br />
Mash the '''Update''' button to save your changed settings.<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
== 8. In Expert Config, reset your titlesDb URL to its original value ==</div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png&diff=2049File:Screenshot-adpnadah-20220708-1135-expert-config-edited-dynamic-url.png2022-07-08T16:38:46Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png&diff=2048File:Screenshot-adpnadah-20220708-1127-expert-config-edit.png2022-07-08T16:29:07Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2047HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:25:36Z<p>CJohnson: /* 1. Pick Your Preservation Node and Get the Peer Code */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
'''WHETHER OR NOT''' you have more than one [[Preservation Node]], you'll need to take down the alphanumeric peer code for the server that you will be using in the test crawl. Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* **You'll need: the Peer Code for your [[Preservation Node]].**<br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
== 8. In Expert Config, reset your titlesDb URL to its original value ==</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2046HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:24:48Z<p>CJohnson: /* 1. Pick Your Preservation Node and Get the Peer Code */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, <code>ADAH</code> for the sole preservation node of the '''Alabama Department of Archives and History'''.) You'll need it below to prepare your titlesDb URL.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* **You'll need: the Peer Code for your [[Preservation Node]].**<br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
== 8. In Expert Config, reset your titlesDb URL to its original value ==</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2045HOWTO: Add a new AU to your node for a test crawl2022-07-08T16:23:49Z<p>CJohnson: /* 1. Pick Your Preservation Node and Get the Peer Code */</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
Make sure to note the alphanumeric code for your preservation node. (For example, `ADAH` for the Alabama Department of Archives and History's sole preservation node.) You'll need it later.<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* **You'll need: the Peer Code for your [[Preservation Node]].**<br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
== 8. In Expert Config, reset your titlesDb URL to its original value ==</div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-20220708-1036-expert-config.png&diff=2044File:Screenshot-adpnadah-20220708-1036-expert-config.png2022-07-08T16:22:48Z<p>CJohnson: CJohnson uploaded a new version of File:Screenshot-adpnadah-20220708-1036-expert-config.png</p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=Peer_Code&diff=2043Peer Code2022-07-08T16:19:04Z<p>CJohnson: Created page with "{| class="wikitable" style="margin:auto" |+ Caption text |- ! Institution !! Institution Code !! Peer Code !! Domain Name |- | Alabama Department of Archives and History || ad..."</p>
<hr />
<div>{| class="wikitable" style="margin:auto"<br />
|+ Caption text<br />
|-<br />
! Institution !! Institution Code !! Peer Code !! Domain Name<br />
|-<br />
| Alabama Department of Archives and History || adah || ADAH || adpnadah.alabama.gov<br />
|-<br />
| Auburn University || aub || AUB ||<br />
|-<br />
| Birmingham Public Library || bpl || BPL ||<br />
|-<br />
| Louisiana State University || lsu || LSU || lsu-liblockss-vm.lsu.edu<br />
|-<br />
| University of Alabama (0) || ua || UAT ||<br />
|-<br />
| University of Alabama (1) || ua || UAT1 ||<br />
|-<br />
| University of North Alabama || una || UNA ||<br />
|}</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_a_test_crawl&diff=2042HOWTO: Add a new AU to your node for a test crawl2022-07-08T15:47:23Z<p>CJohnson: get this page started; we'll come back in a minute</p>
<hr />
<div>:''This HOWTO document is for [[Technical Policy Committee]] members and '''[[Preservation Node]] Managers''' who have been asked to help with the [[test crawl]] for a new [[Archival Unit]] (AU) before it is published to the LOCKSS network for preservation.<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is in the process of being prepared for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]] for the purpose of performing a 1st or 2nd [[test crawl]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Pick Your Preservation Node and Get the Peer Code ==<br />
<br />
'''IF''' your institution operates more than one [[Preservation Node]] on the ADPNet network, you'll only need to pick '''ONE (1)''' node for the test crawl. (If you do a lot of test crawls, you might designate one of your LOCKSS nodes as a dedicated test server, which you use whenever there is a test crawl to be performed.)<br />
<br />
Every preservation node on the ADPNet network has a short alphanumeric code, which we'll call a [[Peer Code]]. The preservation nodes currently on the network are:<br />
<br />
{{:Peer Code}}<br />
<br />
== 2. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 3. Under Expert Config, reset your titleDbs URL to the test feed URL in order to include unpublished AUs ==<br />
<br />
In order to see candidates for [[test crawl]]s, which have not yet been published to the entire network, you'll need to temporarily change a setting in the LOCKSS admin interface that sets the URL for your LOCKSS daemon's titleDbs XML source. <br />
<br />
* **You'll need: the Peer Code for your [[Preservation Node]].**<br />
<br />
[[File:Screenshot-adpnadah-20220708-1036-expert-config.png]]<br />
<br />
== 4. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 5. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 6. Select the institution using "Select AUs" ==<br />
== 7. Select the new AU and mash "Add Selected AUs" ==<br />
== 8. In Expert Config, reset your titlesDb URL to its original value ==</div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-adpnadah-20220708-1036-expert-config.png&diff=2041File:Screenshot-adpnadah-20220708-1036-expert-config.png2022-07-08T15:39:13Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=Main_Page&diff=2040Main Page2022-07-08T15:30:37Z<p>CJohnson: /* HOWTO: Preservation Node Managers */</p>
<hr />
<div>== ALABAMA DIGITAL PRESERVATION NETWORK ==<br />
<br />
* [[Current events|ADPNet Current Events]]<br />
<br />
* [http://www.adpn.org/docs/pdf/ADPNet_Governance_Policy.pdf ADPNet Governance Policy]<br />
* [http://www.adpn.org/docs/pdf/ADPNet_Membership_Model_20111001.pdf ADPNet Membership Model]<br />
* [https://gitlab.com/adpnet ADPNet Gitlab Repository]<br />
<br />
The Alabama Digital Preservation Network (ADPNet) is a distributed digital preservation network, maintained by a number of Alabama libraries, archives, colleges and universities [[The Project|since 2008]]. Member institutions pool their electronic resources through a [[LOCKSS Basics|LOCKSS]]-based network of [[preservation nodes]] in order to provide a long-term storage repository for digital assets. The network can support repositories of different types and sizes as well as different kinds of institutions, and it provides secure, geographically distributed, low-cost, and member-run storage for the long-term preservation of digital assets. ADPNet is governed by the [[Member (ADPNet)|member institutions]] through the network's [[Steering Committee]]; the network is financially supported by [[Member (ADPNet)|member institutions]], and sponsored by the [https://ache.edu/NAAL.aspx Network of Alabama Academic Libraries] (NAAL).<br />
<br />
We welcome new [[Member (ADPNet)|members]] from Alabama and the U.S. Southeast. [http://adpn.org/form.html Contact Us with a Membership Request] if you are interested in joining the digital preservation network.<br />
<br />
=== Committees and Workgroups ===<br />
<br />
* The '''[[Steering Committee]]''' sets general ADPNet policy, reviews and approves requests to join ADPNet, and reviews and approves requests to increase ADPNet storage capacity. Each [[Member (ADPNet)|Member]] of ADPNet appoints voting representatives to the Steering Committee according to [[membership categories]], for a term of 1 year.<br />
<br />
* The '''[[Technical Policy Committee]]''' periodically reviews the ADPNet capacity and technical specifications and prepares recommendations on issues having to do with hardware and software. It consists of members with technical expertise representing the [[ADPNet Host]] member organizations. Chairs of the [[Steering Committee]] serve as ''ex officio'' members of the committee, and appoint regular committee members from the staff of [[ADPNet Host|Host Members]].<br />
<br />
=== Q&A: Network Members ===<br />
* [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]<br />
<br />
=== HOWTO: Network Members ===<br />
* [[HOWTO: Request a staging area on the Drop Server]]<br />
* [[HOWTO: Package files for staging on the Drop Server]]<br />
<br />
==== HOWTO: Preservation Node Managers ====<br />
* [[HOWTO: Add a new AU to your node for preservation]]<br />
* [[HOWTO: Add a new AU to your node for a test crawl]]<br />
* [[HOWTO: Check the version of Java running on your preservation node]]<br />
<br />
== Network History ==<br />
* [[The Project|Project Prospectus]] (February 2008)<br />
<br />
== LOCKSS Resources ==<br />
* [[Getting_Started|Getting Started]] with LOCKSS<br />
** [http://www.lockss.org/support/build-a-lockss-box/ Installation Instructions]<br />
** [http://sourceforge.net/projects/lockss/files/Plugin%20Tool/ Plugin Tool Download] (0.12.1 latest from SourceForge)<br />
** [http://web.archive.org/web/20120201154913/http://www.lockss.org/lockss/Plugin_Tool_Tutorial LOCKSS Plugin Tool Tutorial] (Way Back Machine IA)<br />
** [http://www.adpn.org/docs/pdf/ADPNAnnotation.pdf Annotated ADPNet LOCKSS Plugin]<br />
* Learn more about [[LOCKSS_Software |LOCKSS Software]]<br />
<br />
<br />
<br />
__NOTOC__</div>CJohnsonhttp://www.adpn.org/w/index.php?title=ADPNet_Participant&diff=2039ADPNet Participant2022-05-26T20:45:36Z<p>CJohnson: </p>
<hr />
<div>An [[ADPNet Participant]] is a [[ADPNet Member|Member]] with digital content that needs to be preserved, but without the necessary infrastructure or the desire to host a [[preservation node]]. Participant Members pay a lower annual fee and can purchase storage in smaller allotments than [[ADPNet Host|Host Members]], but pay a higher bulk rate for storage fees, and play a reduced role in the management and governance of the network. Participant members are represented on the [[Steering Committee]] by an elected representative, each one representing up to 5 Participant member institutions. <br />
<br />
== ADPNet Governance Policy ==<br />
<br />
''From [http://www.adpn.org/docs/pdf/ADPNet_Governance_Policy.pdf ADPNet Governance Policy (as revised October 1, 2019)]''<br />
<br />
=== 2.1. Membership Categories ===<br />
<br />
: ADPNet has two categories of membership, each with specific fees and responsibilities. The two categories are:<br />
:<br />
: 1. Host: a category for organizations with the necessary infrastructure to host one or more preservation nodes;<br />
: 2. Participant: a category for organizations with digital content that needs to be preserved, but without the necessary infrastructure or the desire to host a preservation node.<br />
:<br />
: The fees, rights, and responsibilities for each of these membership categories can be found in <br />
Attachment 1: ADPNet Membership Agreement and Attachment 3: ADPNet Membership Model and Storage Fee Schedule. The fee structure is designed to ensure that each Member’s costs are commensurate with its use of the Network.<br />
:<br />
: Membership categories and fees are subject to change. ADPNet Members will be given a six-month advance notice of any upcoming changes in the categories or fees. Membership category is determined at time of joining the network and evaluated annually.<br />
<br />
=== Attachment 1: ADPNet Membership Agreement ===<br />
<br />
: '''PARTICIPANT:'''<br />
:<br />
:* '''Annual membership fee: $300'''<br />
:* '''Storage fees: $25/50GB in 50GB allotments, 50GB minimum allotment'''<br />
:* Governance representation: 1 elected representative/vote per five category members (or portion thereof) on the [[Steering Committee]].<br />
:* Responsibilities:<br />
:** Make material available for ingest by [[Stage content for ingest|staging it on an accessible web server]].<br />
:** Participate in writing [[plug-ins]] for content ingestion.<br />
:** Follow the [[policies and procedures]] of ADPNet.<br />
<br />
[[Category:ADPNet Membership categories]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=ADPNet_Host&diff=2038ADPNet Host2022-05-26T20:45:14Z<p>CJohnson: </p>
<hr />
<div>An '''ADPNet Host''' is a [[ADPNet Member|Member]] organization with the interest and necessary infrastructure and resources to host one or more [[preservation node]]s on the network. Host Members commit to increased membership fees and larger allotments of storage than [[ADPNet Participant|Participant]] members, but pay a lower bulk rate for storage fees. Host Members also appoint representatives directly to the [[Steering Committee]] and [[Technical Policy Committee]].<br />
<br />
== ADPNet Governance Policy ==<br />
<br />
''From [http://www.adpn.org/docs/pdf/ADPNet_Governance_Policy.pdf ADPNet Governance Policy (as revised September 10, 2018)]''<br />
<br />
=== 2.1. Membership Categories ===<br />
<br />
{{:ADPNet Governance 2.1. Membership Categories}}<br />
<br />
=== Attachment 1: ADPNet Membership Agreement ===<br />
<br />
{{:ADPNet Attachment 1. ADPNet Membership Agreement (Host)}}<br />
<br />
<br />
[[Category:ADPNet Membership categories]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Package_files_for_staging_on_the_Drop_Server&diff=2037HOWTO: Package files for staging on the Drop Server2022-05-19T20:30:00Z<p>CJohnson: </p>
<hr />
<div>Here is a checklist based on a write-up by Charles Johnson on 19 February 2021.<br />
<br />
== Check List ==<br />
<br />
Here are some quick practical suggestions on how to package up files into a LOCKSS [[Archival Unit]], and how to get the [[AU]] staged on the [[drop server]] for harvest and consumption by LOCKSS.<br />
<br />
=== First, Start with a Folder ===<br />
<br />
'''STEP 1.''' The materials you will be preserving '''SHOULD''' start out as files packaged into a directory tree under one top-level folder.<br />
<br />
* '''FORMAT:''' The directory can be organized into any hierarchical structure of folders and subfolders you want, so long as it’s all stored underneath one top-level folder.<br />
* '''EXAMPLE:''' ADAH is preserving high-quality digital masters from our ongoing scanning projects in a set of directories called Q-Numbers Masters files. Each directory contains 500 TIFF files (with poetic and evocative names like <code>Q0000150001.tif</code>), along with some additional files for metadata pertaining to that package of files. When we start out, the directory that we're going to package for upload looks like this:<br />
<br />
Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m\<br />
- Q0000150001.tif<br />
- Q0000150002.tif<br />
- Q0000150003.tif<br />
[…]<br />
- Q0000150500.tif<br />
<br />
* '''EXCEPTION:''' If any of the files in your subdirectories happen to be named <code>index.html</code> or <code>index.htm</code>, this is a special case that requires certain precautions to make sure that all your content gets correctly harvested. ''Check with the ADPNet TPC for details on how to deal with this.''<br />
* '''RECOMMENDATION:''' The number and size of files is up to you, but there are some practical constraints based on network capacity. Best practice is to divvy up assets into AUs that will contain '''LESS THAN 1,000,000 or so individual files,''' and '''LESS THAN 1 TiB of data''' per AU. The LOCKSS network can in fact handle very, very large AUs, and ADPNet is currently preserving AUs that are larger than these suggested, fairly arbitrary limits. But (1) nodes like that take forever to upload and crawl, which means that it’s a much slower turnaround time for you before we can confirm that they are preserved in the network; and (2) nodes like that also make some practical systems administration tasks more of a pain in the neck for the people who run the [[preservation nodes]].<br />
<br />
=== Second, Name the Folder ===<br />
<br />
'''STEP 2.''' Your top-level folder can have almost any name, but the name '''MUST''' be unique among all the AUs you will ever upload.<br />
<br />
* Once you ingest an AU, you '''SHOULD NOT''' re-use that directory name unless you actually intend to replace the old materials with the new materials. You need a new name to ingest new AUs.<br />
<br />
* '''EXAMPLE:''' ADAH has a bunch of Q-Number Masters files to stage for ingest, so we give the directory a name unique to those contents, for example: Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m (so, next time we upload materials, we upload them under a new directory with the next numbers in the sequence, Digitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m).<br />
<br />
=== Third, Bag the Folder ===<br />
<br />
'''STEP 3.''' Once you have your top-level folder prepared and named, you '''SHOULD''' enclose the folder in a BagIt formatted directory.<br />
<br />
* You can do this easily using an open-source Python script ([https://github.com/LibraryOfCongress/bagit-python BagIt-Python]) or using the Bagger application ([https://github.com/LibraryOfCongress/bagger Bagger]).<br />
<br />
* '''EXAMPLE:''' When a Q-numbers directory is ready to be bagged at ADAH, I open Windows PowerShell, then I run:<br />
<br />
python bagit.py ${DIRNAME}<br />
<br />
* This encloses the directory with BagIt preservation data. As a result, <code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m </code>(for example) is now reorganized so that the top-level folder contains a single “payload” subdirectory, called <code>data</code>, that contains the 500+ TIFFs and associated metadata files, and then a set of small text files (<code>baginfo.txt</code>, <code>bagit.txt</code>, <code>manifest-sha256.txt</code>, <code>tag-manifest-sha256.txt</code>, etc.) that provide a manifest and checksums for those payload files, along with some meta-data about the packaging process.<br />
* If I want to validate the contents of the preservation package before I upload it, I can do that by running this on the same directory:<br />
<br />
python bagit.py --validate ${DIRNAME}<br />
<br />
=== Fourth, Prepare a LOCKSS Manifest and Drop It Into the Top Level of the Bag ===<br />
<br />
'''STEP 4.''' Once you have your top-level folder prepared, named, and bagged, you '''MUST''' create a small HTML file named <code>manifest.html</code> and drop it into the top-level directory alongside <code>baginfo.txt</code>, <code>bagit.txt</code>, etc. The required format for this manifest file is very simple, but the best way to create it is still to use a tool like the [http://adpn.org/services/MakeManifest/ ADAH Make Manifest web form] or the [https://gitlab.com/adpnet/adpn-cli adpn-cli command-line tool].<br />
<br />
* '''FORMAT:''' The <code>manifest.html</code> file needs to include a link to your AU’s location on the drop server, some boilerplate HTML, and some boilerplate language that gives the LOCKSS daemon permission to harvest content. This is a bit of a pain in the neck and the format is under-documented, but LOCKSS won’t ingest your AU unless it includes a file like this with the correct URL and the correct boilerplate language.<br />
* '''EXAMPLE:''' After I’ve bagged a Q-Numbers directory, I generate a <code>manifest.html</code> file using a script to file in a standard template with information about the [[AU]] I’m about to upload. I place the file in the top level of the BagIt-formatted directory, alongside <code>baginfo.txt</code>, <code>bagit.txt</code>, etc. So now my directory looks like:<br />
<br />
Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m\<br />
- bagit.txt<br />
- baginfo.txt<br />
- manifest.html<br />
- manifest-sha256.txt<br />
- manifest-sha512.txt<br />
- tagmanifest-sha256.txt<br />
- tagmanifest-sha512.txt<br />
- data\<br />
- Q0000150001.tif<br />
- Q0000150002.tif<br />
- Q0000150003.tif<br />
[…]<br />
- Q0000150500.tif<br />
<br />
* '''RECOMMENDATION:''' You can access a version of the templates I use if you go to this URL:<br />
<br />
[http://adpn.org/services/MakeManifest/ adpn.org/services/MakeManifest/]<br />
<br />
* Fill in the form fields with your own information. Some notes:<br />
** The "Institution Code" is the username you’ve been assigned (for example, <code>adah</code> or <code>tsk</code>)<br />
** The "Institution Name" is the human-readable name for your institution, possibly followed by an alphanumeric code used by ADPNet (for example "Alabama Department of Archives and History (ADAH)"). If your institution is not listed as a recognized publisher of AUs, contact ADPNet TPC to be added.<br />
** The "Directory Name" is the unique name you chose in step 2.<br />
** The "File Size" field should contain the human-readable summary information that you would get from <code>File > Properties</code> or from a command-line tool like <code>du</code>. Ideally, this should include both the total cumulative size of all the files in the AU (in units like MiB, GiB, TiB, ...) and also the total count of files in the AU.<br />
** The "Staging Area Base URL" and the "LOCKSS Plugin" fields are pre-filled with standard default values, which should not need to be changed as long as you are working with our drop server.<br />
* Use the '''CREATE FILE''' button to generate HTML for your manifest file. You can save the results using <code>File > Save</code> or by copying and pasting the HTML source code into a text editor. Make sure the results are saved as <code>manifest.html</code> and then place that file in the top level of the BagIt-formatted directory, alongside <code>baginfo.txt</code>, <code>bagit.txt</code>, etc.<br />
<br />
=== Fifth, Upload Your Archival Unit to Your Drop Server Staging Area ===<br />
<br />
'''STEP 5.''' At this point your AU should be packaged and ready to be considered as an [[Archival Unit]] (AU) by the LOCKSS daemon.<br />
<br />
* Use WinSCP (or any other SFTP tool that you like) to upload the whole packaged-up directory to <code>drop.adpn.org</code>, storing it under the <code>drop_au_content_in_here</code> subdirectory of your staging area.<br />
* Notify ADPNet TPC to let us know you’re ready to go ahead with the ingest.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Stage_content_for_ingest&diff=2033Stage content for ingest2022-04-12T15:19:37Z<p>CJohnson: </p>
<hr />
<div>To preserve digital content in ADPNet, the content must be [[Stage content for ingest|staged for ingest]]. That means making it available on a web server. The staged content should be accessible to ADPNet [[preservation node]]s; it does ''not'' need to be available to the public web. ''Staging content'' for ADPNet servers to [[harvest]] is the method by which content is ingested into the digital preservation network.<br />
<br />
== Procedure ==<br />
'''''This section is now obsolete (as of April 2022). ADAH uses, and ADPNet strongly recommends, the use of the ADPNet [[Drop Server]] for staging content for preservation. This server is available to all network members. Documentation should be updated soon.'''''<br />
<br />
The best way to stage content currently depends on the details of your local web server setup. Here's how we do it at the [http://archives.alabama.gov/ Alabama Department of Archives and History] as of June 2019:<br />
<br />
1. We have a public web server, [http://archives.alabama.gov/ archives.alabama.gov]. There is a subdirectory provided by this web server at [http://archives.alabama.gov/Lockss/ http://archives.alabama.gov/Lockss/] which is set up to be visible only to the IP addresses of LOCKSS preservation nodes on ADPNet. Anyone else who browses to that subdirectory will see an HTTP 403 Forbidden error. (We used IIS's access rules to set up limited access to this directory. If you need help setting up a limited-access subdirectory on your own web server, contact the [[Technical Policy Committee]] for assistance.<br />
<br />
2. New content is staged as a subdirectory uploaded to [http://archives.alabama.gov/Lockss/ http://archives.alabama.gov/Lockss/] . We use FTP to create a new subdirectory on the web server, for example [http://archives.alabama.gov/Lockss/WPA-Folder-01 http://archives.alabama.gov/Lockss/WPA-Folder-01], and then transfer preservation copies of all of the content for that Archival Unit to the subdirectory. Files can be any format and can have any name, as long as none of them is called index, index.html, or any other file name with "index" before the extension (for example: don't use this technique to stage a file named index.php, index.htm, index.asp, index.cfm, ...).<br />
<br />
3. Next, we upload a [[manifest file]] named manifestpage.html to the same subdirectory, which includes a permissions statement and a link to the directory listing (for example [http://archives.alabama.gov/Lockss/WPA-Folder-01 http://archives.alabama.gov/Lockss/WPA-Folder-01]).<br />
<br />
4. Once the manifest file is uploaded and the AU is accessible through the web server, the content is now considered staged for ingest.<br />
<br />
<br />
== See also ==<br />
[[Local Content Staging (ADPNet Model.docx)]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Unique_Name&diff=2032Unique Name2022-04-12T15:15:46Z<p>CJohnson: </p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''SHOULD''' be organized into a single top-level directory with a [[Unique Name]].<br />
<br />
This will typically be the directory name for the AU on the [[Drop Server]]. If you are using the Drop Server then your top-level folder can have almost any name, but that name '''MUST''' be unique among all the AUs you will ever upload.<br />
<br />
* Once you ingest an AU, you '''SHOULD NOT''' re-use that directory name unless you actually intend to ''replace'' the old materials the with new materials. You need a new name to ingest new AUs.<br />
** '''EXAMPLE:''' ADAH has a bunch of Q-Number Masters files to stage for ingest. Each of these is a package of 500 large TIFF files produced by our Digitization section. So we give the directory a name unique to those contents, and based on the range of internal identifier numbers ("Q Numbers") for those master files. For example: <code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code> contains the 500 Master files with identifiers Q0000150001-Q0000150500; next time we upload newly-created materials, we will upload them under a new directory with the next numbers in the sequence, to the directory <code>Digitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>.<br />
<br />
* Typically AUs '''SHOULD''' be organized in order to ensure [[Immutable Content]], but if you end up in circumstances where you are required to add, remove, or correct content within an AU, you should upload the updated materials to a directory on the Drop Server with '''exactly the same directory name''' as you used for that AU when it was first ingested.<br />
** LOCKSS content harvesters treat content as an update to an existing AU '''if, and only if,''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
** A collection '''does not need to have been stored continuously''' on the staging server in order to update it.<br />
** In that case, make sure to upload ''all'' the content of the updated not just the new or modified content -- the AU will be entirely replaced by the corrected AU.<br />
<br />
* When you unstage an [[AU]] from the staging server, we '''strongly recommend''' that you retain the uniquely-named directory on the server. You should remove the LOCKSS [[Manifest Page]] and empty the directory, replacing it with a small plain-text README file indicating that the directory was previously used to stage an AU for ingest.<br />
** This allows for easy tracking of the names you have used over time, which will make it much easier for you and for automated tools to guarantee [[Unique Name]]s even after you have preserved dozens or hundreds of AUs.<br />
** This also provides a ready target directory if you should need to stage an update to the AU under exceptional circumstances.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Unique_Name&diff=2031Unique Name2022-04-12T15:15:30Z<p>CJohnson: </p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''SHOULD''' be organized into a single top-level directory with a [[Unique Name]].<br />
<br />
This will typically be the directory name for the AU on the [[Drop Server]]. If you are using the Drop Server then your top-level folder can have almost any name, but that name '''MUST''' be unique among all the AUs you will ever upload.<br />
<br />
* Once you ingest an AU, you '''SHOULD NOT''' re-use that directory name unless you actually intend to ''replace''' the old materials the with new materials. You need a new name to ingest new AUs.<br />
** '''EXAMPLE:''' ADAH has a bunch of Q-Number Masters files to stage for ingest. Each of these is a package of 500 large TIFF files produced by our Digitization section. So we give the directory a name unique to those contents, and based on the range of internal identifier numbers ("Q Numbers") for those master files. For example: <code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code> contains the 500 Master files with identifiers Q0000150001-Q0000150500; next time we upload newly-created materials, we will upload them under a new directory with the next numbers in the sequence, to the directory <code>Digitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>.<br />
<br />
* Typically AUs '''SHOULD''' be organized in order to ensure [[Immutable Content]], but if you end up in circumstances where you are required to add, remove, or correct content within an AU, you should upload the updated materials to a directory on the Drop Server with '''exactly the same directory name''' as you used for that AU when it was first ingested.<br />
** LOCKSS content harvesters treat content as an update to an existing AU '''if, and only if,''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
** A collection '''does not need to have been stored continuously''' on the staging server in order to update it.<br />
** In that case, make sure to upload ''all'' the content of the updated not just the new or modified content -- the AU will be entirely replaced by the corrected AU.<br />
<br />
* When you unstage an [[AU]] from the staging server, we '''strongly recommend''' that you retain the uniquely-named directory on the server. You should remove the LOCKSS [[Manifest Page]] and empty the directory, replacing it with a small plain-text README file indicating that the directory was previously used to stage an AU for ingest.<br />
** This allows for easy tracking of the names you have used over time, which will make it much easier for you and for automated tools to guarantee [[Unique Name]]s even after you have preserved dozens or hundreds of AUs.<br />
** This also provides a ready target directory if you should need to stage an update to the AU under exceptional circumstances.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Unique_Name&diff=2030Unique Name2022-04-12T15:15:10Z<p>CJohnson: </p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''MUST''' be organized into a single top-level directory with a [[Unique Name]].<br />
<br />
This will typically be the directory name for the AU on the [[Drop Server]]. Your top-level folder can have almost any name, but the name '''MUST''' be unique among all the AUs you will ever upload.<br />
<br />
* Once you ingest an AU, you '''SHOULD NOT''' re-use that directory name unless you actually intend to ''replace''' the old materials the with new materials. You need a new name to ingest new AUs.<br />
** '''EXAMPLE:''' ADAH has a bunch of Q-Number Masters files to stage for ingest. Each of these is a package of 500 large TIFF files produced by our Digitization section. So we give the directory a name unique to those contents, and based on the range of internal identifier numbers ("Q Numbers") for those master files. For example: <code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code> contains the 500 Master files with identifiers Q0000150001-Q0000150500; next time we upload newly-created materials, we will upload them under a new directory with the next numbers in the sequence, to the directory <code>Digitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>.<br />
<br />
* Typically AUs '''SHOULD''' be organized in order to ensure [[Immutable Content]], but if you end up in circumstances where you are required to add, remove, or correct content within an AU, you should upload the updated materials to a directory on the Drop Server with '''exactly the same directory name''' as you used for that AU when it was first ingested.<br />
** LOCKSS content harvesters treat content as an update to an existing AU '''if, and only if,''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
** A collection '''does not need to have been stored continuously''' on the staging server in order to update it.<br />
** In that case, make sure to upload ''all'' the content of the updated not just the new or modified content -- the AU will be entirely replaced by the corrected AU.<br />
<br />
* When you unstage an [[AU]] from the staging server, we '''strongly recommend''' that you retain the uniquely-named directory on the server. You should remove the LOCKSS [[Manifest Page]] and empty the directory, replacing it with a small plain-text README file indicating that the directory was previously used to stage an AU for ingest.<br />
** This allows for easy tracking of the names you have used over time, which will make it much easier for you and for automated tools to guarantee [[Unique Name]]s even after you have preserved dozens or hundreds of AUs.<br />
** This also provides a ready target directory if you should need to stage an update to the AU under exceptional circumstances.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Unique_Name&diff=2029Unique Name2022-04-12T15:14:58Z<p>CJohnson: Created page with "Wherever possible, the Archival Units (AUs) that you preserve in ADPNet '''MUST''' be organized into a single top-level directory with a Unique Name. This will typica..."</p>
<hr />
<div>Wherever possible, the [[Archival Units]] (AUs) that you preserve in ADPNet '''MUST''' be organized into a single top-level directory with a [[Unique Name]].<br />
<br />
This will typically be the directory name for the AU on the [[Drop Server]]. Your top-level folder can have almost any name, but the name '''MUST''' be unique among all the AUs you will ever upload.<br />
<br />
* Once you ingest an AU, you '''SHOULD NOT''' re-use that directory name unless you actually intend to ''replace''' the old materials the with new materials. You need a new name to ingest new AUs.<br />
** '''EXAMPLE:''' ADAH has a bunch of Q-Number Masters files to stage for ingest. Each of these is a package of 500 large TIFF files produced by our Digitization section. So we give the directory a name unique to those contents, and based on the range of internal identifier numbers ("Q Numbers") for those master files. For example: <code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code> contains the 500 Master files with identifiers Q0000150001-Q0000150500; next time we upload newly-created materials, we will upload them under a new directory with the next numbers in the sequence, to the directory <code>Digitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>.<br />
<br />
* Typically AUs '''SHOULD''' be organized in order to ensure [[Immutable Content]], but if you end up in circumstances where you are required to add, remove, or correct content within an AU, you should upload the updated materials to a directory on the Drop Server with '''exactly the same directory name''' as you used for that AU when it was first ingested.<br />
** LOCKSS content harvesters treat content as an update to an existing AU '''if, and only if,''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
** A collection '''does not need to have been stored continuously''' on the staging server in order to update it.<br />
** In that case, make sure to upload ''all'' the content of the updated not just the new or modified content -- the AU will be entirely replaced by the corrected AU.<br />
<br />
* When you unstage an [[AU]] from the staging server, we '''strongly recommend''' that you retain the uniquely-named directory on the server. You should remove the LOCKSS [[Manifest Page]] and empty the directory, replacing it with a small plain-text README file indicating that the directory was previously used to stage an AU for ingest.<br />
** This allows for easy tracking of the names you have used over time, which will make it much easier for you and for automated tools to guarantee [[Unique Name]]s even after you have preserved dozens or hundreds of AUs.<br />
** This also provides a ready target directory if you should need to stage an update to the AU under exceptional circumstances.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Immutable_Content&diff=2028Immutable Content2022-04-12T15:04:41Z<p>CJohnson: </p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''SHOULD''' have [[Immutable Content]]:<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AU]]s in such a way that files '''DO NOT''' need to be added to or deleted from the collection.<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AU]]s in such a way that the content of files in the AU '''DOES NOT''' need to be updated over time.<br />
<br />
If you intend to preserve a digital collection that grows or changes over time, ADPNet '''strongly recommends''' that you adopt a packaging scheme that arranges the items in series, and subdivides the collection into consistent chunks.<br />
<br />
# Items within a growing collection '''MAY''' be arranged into AUs of fixed size based on identifiers within a series.<br />
#* '''EXAMPLE #1.''' ADAH Digitization Masters are arranged by their internal unique identifier numbers, and then divvied up into blocks of 500 Master files each. Master files are large TIFF image files, each of which is assigned an internal "Q-Number" in sequence, for example <code>Q0000150001.tif</code>, <code>Q0000150002.tif</code>, <code>Q0000150003.tif</code>, and so on. We package these up into directories of 500 TIFF files each, which are assigned [[Unique Name]]s based on the range of Q-number identifiers (<code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code>, <code>igitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>, and so on.)<br />
# Items within a growing collection '''MAY''' be arranged into AUs based on a timestamp if a fixed size doesn't fit your production schedule well.<br />
#* '''EXAMPLE #2.''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is impractical to wait until a large number of items are filled before packaging content for preservation. Instead, items within the digital repository are organized into blocks based on a fixed timestamp, in this case the month in which they were produced or uploaded to the server. AUs have names like '''repository-2020-11''', '''repository-2020-12''', and so on, based on a directory structure that organizes files into date-based directories like <code>2020/11</code>, <code>2020/12</code>, and so on.<br />
<br />
=== Why? ===<br />
<br />
* Digital preservation is much easier to do correctly &#8212; and it is much easier to prove that it has been done correctly &#8212; when the units of preservation are immutable.<br />
<br />
* Practically, the design of the LOCKSS software makes it easier to achieve, maintain, and prove consensus on the contents of AUs if the contents of AUs do not change over time.<br />
<br />
* Practically, other components that we (ADPNet) use in our [[HOWTO: Package files for staging on the Drop Server|packaging guidelines]] (for example, BagIt) also depend on [[checksum]]s and fixity checks for validation; these will work best and require the least troublesome work when applied to immutable units of data rather than changing data sets.<br />
<br />
=== What if I absolutely, positively need to change the content of an AU? ===<br />
<br />
Sometimes stuff happens. The LOCKSS daemon '''does''' provide the capability to version content. We strongly discourage relying on this capability routinely, and strongly encourage organizing [[AU]]s so as to avoid it, but if you end up in a situation where you just ''have to'' add or remove content for reasons of correctness or due to institutional obligations, then check out the instructions in our answer for [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Immutable_Content&diff=2027Immutable Content2022-04-12T15:04:09Z<p>CJohnson: </p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''SHOULD''' have [[Immutable Content]]:<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AU]]s in such a way that files '''DO NOT''' need to be added to or deleted from the collection.<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AUs]] in such a way that the content of files in the AU '''DOES NOT''' need to be updated over time.<br />
<br />
If you intend to preserve a digital collection that grows or changes over time, ADPNet '''strongly recommends''' that you adopt a packaging scheme that arranges the items in series, and subdivides the collection into consistent chunks.<br />
<br />
# Items within a growing collection '''MAY''' be arranged into AUs of fixed size based on identifiers within a series.<br />
#* '''EXAMPLE #1.''' ADAH Digitization Masters are arranged by their internal unique identifier numbers, and then divvied up into blocks of 500 Master files each. Master files are large TIFF image files, each of which is assigned an internal "Q-Number" in sequence, for example <code>Q0000150001.tif</code>, <code>Q0000150002.tif</code>, <code>Q0000150003.tif</code>, and so on. We package these up into directories of 500 TIFF files each, which are assigned [[Unique Name]]s based on the range of Q-number identifiers (<code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code>, <code>igitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>, and so on.)<br />
# Items within a growing collection '''MAY''' be arranged into AUs based on a timestamp if a fixed size doesn't fit your production schedule well.<br />
#* '''EXAMPLE #2.''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is impractical to wait until a large number of items are filled before packaging content for preservation. Instead, items within the digital repository are organized into blocks based on a fixed timestamp, in this case the month in which they were produced or uploaded to the server. AUs have names like '''repository-2020-11''', '''repository-2020-12''', and so on, based on a directory structure that organizes files into date-based directories like <code>2020/11</code>, <code>2020/12</code>, and so on.<br />
<br />
=== Why? ===<br />
<br />
* Digital preservation is much easier to do correctly &#8212; and it is much easier to prove that it has been done correctly &#8212; when the units of preservation are immutable.<br />
<br />
* Practically, the design of the LOCKSS software makes it easier to achieve, maintain, and prove consensus on the contents of AUs if the contents of AUs do not change over time.<br />
<br />
* Practically, other components that we (ADPNet) use in our [[HOWTO: Package files for staging on the Drop Server|packaging guidelines]] (for example, BagIt) also depend on [[checksum]]s and fixity checks for validation; these will work best and require the least troublesome work when applied to immutable units of data rather than changing data sets.<br />
<br />
=== What if I absolutely, positively need to change the content of an AU? ===<br />
<br />
Sometimes stuff happens. The LOCKSS daemon '''does''' provide the capability to version content. We strongly discourage relying on this capability routinely, and strongly encourage organizing [[AU]]s so as to avoid it, but if you end up in a situation where you just ''have to'' add or remove content for reasons of correctness or due to institutional obligations, then check out the instructions in our answer for [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Immutable_Content&diff=2026Immutable Content2022-04-12T15:03:29Z<p>CJohnson: /* Why? */</p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''SHOULD HAVE''' [[Immutable Content]]:<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AU]]s in such a way that files '''DO NOT''' need to be added to or deleted from the collection.<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AUs]] in such a way that the content of files in the AU '''DOES NOT''' need to be updated over time.<br />
<br />
If you intend to preserve a digital collection that grows or changes over time, ADPNet '''strongly recommends''' that you adopt a packaging scheme that arranges the items in series, and subdivides the collection into consistent chunks.<br />
<br />
# Items within a growing collection '''MAY''' be arranged into AUs of fixed size based on identifiers within a series.<br />
#* '''EXAMPLE #1.''' ADAH Digitization Masters are arranged by their internal unique identifier numbers, and then divvied up into blocks of 500 Master files each. Master files are large TIFF image files, each of which is assigned an internal "Q-Number" in sequence, for example <code>Q0000150001.tif</code>, <code>Q0000150002.tif</code>, <code>Q0000150003.tif</code>, and so on. We package these up into directories of 500 TIFF files each, which are assigned [[Unique Name]]s based on the range of Q-number identifiers (<code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code>, <code>igitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>, and so on.)<br />
# Items within a growing collection '''MAY''' be arranged into AUs based on a timestamp if a fixed size doesn't fit your production schedule well.<br />
#* '''EXAMPLE #2.''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is impractical to wait until a large number of items are filled before packaging content for preservation. Instead, items within the digital repository are organized into blocks based on a fixed timestamp, in this case the month in which they were produced or uploaded to the server. AUs have names like '''repository-2020-11''', '''repository-2020-12''', and so on, based on a directory structure that organizes files into date-based directories like <code>2020/11</code>, <code>2020/12</code>, and so on.<br />
<br />
=== Why? ===<br />
<br />
* Digital preservation is much easier to do correctly &#8212; and it is much easier to prove that it has been done correctly &#8212; when the units of preservation are immutable.<br />
<br />
* Practically, the design of the LOCKSS software makes it easier to achieve, maintain, and prove consensus on the contents of AUs if the contents of AUs do not change over time.<br />
<br />
* Practically, other components that we (ADPNet) use in our [[HOWTO: Package files for staging on the Drop Server|packaging guidelines]] (for example, BagIt) also depend on [[checksum]]s and fixity checks for validation; these will work best and require the least troublesome work when applied to immutable units of data rather than changing data sets.<br />
<br />
=== What if I absolutely, positively need to change the content of an AU? ===<br />
<br />
Sometimes stuff happens. The LOCKSS daemon '''does''' provide the capability to version content. We strongly discourage relying on this capability routinely, and strongly encourage organizing [[AU]]s so as to avoid it, but if you end up in a situation where you just ''have to'' add or remove content for reasons of correctness or due to institutional obligations, then check out the instructions in our answer for [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Immutable_Content&diff=2025Immutable Content2022-04-12T15:03:00Z<p>CJohnson: Created page with "Wherever possible, the Archival Units (AUs) that you preserve in ADPNet '''SHOULD HAVE''' Immutable Content: * Network Members '''SHOULD''' package digital preservati..."</p>
<hr />
<div>Wherever possible, the [[Archival Unit]]s (AUs) that you preserve in ADPNet '''SHOULD HAVE''' [[Immutable Content]]:<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AU]]s in such a way that files '''DO NOT''' need to be added to or deleted from the collection.<br />
<br />
* Network Members '''SHOULD''' package digital preservation content into [[AUs]] in such a way that the content of files in the AU '''DOES NOT''' need to be updated over time.<br />
<br />
If you intend to preserve a digital collection that grows or changes over time, ADPNet '''strongly recommends''' that you adopt a packaging scheme that arranges the items in series, and subdivides the collection into consistent chunks.<br />
<br />
# Items within a growing collection '''MAY''' be arranged into AUs of fixed size based on identifiers within a series.<br />
#* '''EXAMPLE #1.''' ADAH Digitization Masters are arranged by their internal unique identifier numbers, and then divvied up into blocks of 500 Master files each. Master files are large TIFF image files, each of which is assigned an internal "Q-Number" in sequence, for example <code>Q0000150001.tif</code>, <code>Q0000150002.tif</code>, <code>Q0000150003.tif</code>, and so on. We package these up into directories of 500 TIFF files each, which are assigned [[Unique Name]]s based on the range of Q-number identifiers (<code>Digitization-Masters-Q-numbers-Master-Q0000150001_Q0000150500m</code>, <code>igitization-Masters-Q-numbers-Master-Q0000105501_Q0000106000m</code>, and so on.)<br />
# Items within a growing collection '''MAY''' be arranged into AUs based on a timestamp if a fixed size doesn't fit your production schedule well.<br />
#* '''EXAMPLE #2.''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is impractical to wait until a large number of items are filled before packaging content for preservation. Instead, items within the digital repository are organized into blocks based on a fixed timestamp, in this case the month in which they were produced or uploaded to the server. AUs have names like '''repository-2020-11''', '''repository-2020-12''', and so on, based on a directory structure that organizes files into date-based directories like <code>2020/11</code>, <code>2020/12</code>, and so on.<br />
<br />
=== Why? ===<br />
<br />
* Digital preservation is much easier to do correctly &#8212; and it is much easier to prove that it has been done correctly &#8212; when the units of preservation are immutable.<br />
<br />
* Practically, the design of the LOCKSS software makes it easier to achieve, maintain, and prove consensus on the contents of AUs if the contents of AUs do not change over time.<br />
<br />
* Practically, other components that we (ADPNet) use in our [[HOWTO: Package files for staging on the Drop Server|packaging guidelines]] (for example, BagIt) also depend on [[checksums]] and fixity checks for validation; these will work best and require the least troublesome work when applied to immutable units of data rather than changing data sets.<br />
<br />
=== What if I absolutely, positively need to change the content of an AU? ===<br />
<br />
Sometimes stuff happens. The LOCKSS daemon '''does''' provide the capability to version content. We strongly discourage relying on this capability routinely, and strongly encourage organizing [[AU]]s so as to avoid it, but if you end up in a situation where you just ''have to'' add or remove content for reasons of correctness or due to institutional obligations, then check out the instructions in our answer for [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=BagIt&diff=2024BagIt2022-04-12T14:39:35Z<p>CJohnson: Redirected page to HOWTO: Package files for staging on the Drop Server#Third.2C Bag the Folder</p>
<hr />
<div>#REDIRECT [[HOWTO: Package files for staging on the Drop_Server#Third.2C_Bag_the_Folder]]</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Archival_Unit&diff=2023Archival Unit2022-04-12T14:38:51Z<p>CJohnson: </p>
<hr />
<div>An [[Archival Unit]] (AU) is the basic unit of preservation on the ADPNet LOCKSS network. Each AU has a [[Manifest Page]] and a collection of digital assets that the manifest page links to.<br />
<br />
ADPNet strongly recommends that AUs to be preserved in our network '''SHOULD''' be organized into a single top-level directory with a [[Unique Name]] and [[Immutable Content]]. (If you want to preserve items from a collection that grows over time, [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?|we strongly recommend that you arrange the material in series and subdivide it into multiple AUs]] based on identifier numbers or fixed dates.) The digital contents for preservation '''SHOULD''' be [[BagIt|enclosed within a BagIt-formatted directory]] to help validate transmission and preservation.<br />
<br />
== Network Members ==<br />
* [[HOWTO: Package files for staging on the Drop Server]]<br />
* [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]<br />
<br />
== Preservation Node Managers ==<br />
<br />
* [[HOWTO: Add a new AU to your node for preservation]]<br />
<br />
== Technical Documentation ==<br />
=== From [https://plnwiki.lockss.org/index.php?title=LOCKSS_Technical_Manual LOCKSS Technical Manual] ===<br />
<br />
: LOCKSS caches: The LOCKSS caches are the nodes in the network in which digital objects will be preserved and monitored as archival units. One archival unit is typically a one-year collection (size: 1GB to 20GB). The size of an AU results of a trade-off between the processing overhead required by large AUs and the guarantee that all AUs integrity can be regularly checked in the case of a multitude of small AUs. The daemon is a java application which collects digital objects through http requests from the original website, store them inside the cache as an Archival Unit, computes an SHA-1 checksum and regularly monitors their integrity by comparing the preserved content with the other caches in the network with a specific voting protocol (LCAP). The AUs are collected at various moments in time by the different nodes in the network to reduce the risk of communication issues. The content is regularly recrawled from the original website if it is still available. If a new version of the AU is available, the previous version is kept but only the most recent AUs will be checked for fixity. <br />
:<br />
: &#8212;From [https://plnwiki.lockss.org/index.php?title=LOCKSS_Technical_Manual LOCKSS Technical Manual: Basic Private LOCKSS Network infrastructure] (Plnwiki)<br />
<br />
=== From [[Getting Started with LOCKSS, Midge Coates, 2013-06|"Getting Started with LOCKSS" (June 2016)]] ===<br />
<br />
: Each [[Archival Unit]] (AU) must be Web-accessible so the [[LOCKSS daemon]] can get at it. That means that it needs to be put on a Web-accessible server computer. If you have a server, but it isn't Web-accessible or access to it is blocked (at the firewall, for example), the LOCKSS crawl won't work. You can use a firewall to protect your collection, but make sure the LOCKSS daemon has access. Check with your IT support person or system administrator to confirm that this is the case.<br />
:<br />
: [...]<br />
:<br />
: The next major part of getting collections ingested is preparing each collection’s [[manifest page]]. The manifest page is a basic HTML document that contains at least two things: the permission statement that gives the LOCKSS daemon the right to harvest the collection (at the foot of the document) and the base URL of the AU for the collection (or for a fraction of the collection).<br />
:<br />
: Here is an example of a fairly simple manifest page for a fairly simple collection (Auburn University’s Alabama Postcards images collection). Feel free to use it as a template. <br />
:<br />
: [[Image:ExampleAlabamaPostcardsCollectionManifest.png]]<br />
:<br />
: The base URL isn't visible if you've opened the manifest in a browser window, because it's located inside the HTML <a href> tag. To see this tag, open the HTML file in DreamWeaver or Notepad++ . If you're opening it in a browser window, go to View => Source to see the tags.<br />
:<br />
: I usually put all the URLs for the sub-folders in my AU, but the LOCKSS folks tell me it isn't really necessary, as the daemon will go to all the folders inside the original unless you tell it not to. So you could leave out everything except the base URL. NOTE: These URLS are '''not''' the CONTENTdm collection URLs; they're the URLs for the collections (or AUs) on your Web server.<br />
:<br />
: The manifest page should be kept in the same folder as the AU so that the LOCKSS daemon can find it easily. If you break a collection up into digestible chunks (50 GB or so), you should have a separate manifest file for each AU chunk, with each AU chunk in a separate folder, along with its own manifest page. I also like to do a metadata export from the relevant CONTENTdm collection as a tab-delimited text file and put a copy of that in each folder also, in case the CONTENTdm collection needs to be reconstituted after a disaster.<br />
:<br />
: So, if you have one AU for a collection, you need one manifest with the appropriate base URL. If you have two AUs, you need two manifests, one in each AU folder, each with the appropriate base URL for that particular AU. Here is a screenshot of the folder directory for the Alabama Postcards AU, so you can see what is in the folder. The manifest has a little box around it, so you can't miss it.<br />
:<br />
: [[Image:ExampleAlabamaPostcardsCollectionDirectoryListing.png]]<br />
:<br />
:<br />
: &#8212;From "[[Getting Started with LOCKSS, Midge Coates, 2013-06|Getting Started with LOCKSS]]" (June 2016) by Midge Coates</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Archival_Unit&diff=2022Archival Unit2022-04-12T14:29:24Z<p>CJohnson: </p>
<hr />
<div>An [[Archival Unit]] (AU) is the basic unit of preservation on the ADPNet LOCKSS network. Each AU has a [[Manifest Page]] and a collection of digital assets that the manifest page links to.<br />
<br />
== Network Members ==<br />
* [[HOWTO: Package files for staging on the Drop Server]]<br />
* [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]<br />
<br />
== Preservation Node Managers ==<br />
<br />
* [[HOWTO: Add a new AU to your node for preservation]]<br />
<br />
== Technical Documentation ==<br />
=== From [https://plnwiki.lockss.org/index.php?title=LOCKSS_Technical_Manual LOCKSS Technical Manual] ===<br />
<br />
: LOCKSS caches: The LOCKSS caches are the nodes in the network in which digital objects will be preserved and monitored as archival units. One archival unit is typically a one-year collection (size: 1GB to 20GB). The size of an AU results of a trade-off between the processing overhead required by large AUs and the guarantee that all AUs integrity can be regularly checked in the case of a multitude of small AUs. The daemon is a java application which collects digital objects through http requests from the original website, store them inside the cache as an Archival Unit, computes an SHA-1 checksum and regularly monitors their integrity by comparing the preserved content with the other caches in the network with a specific voting protocol (LCAP). The AUs are collected at various moments in time by the different nodes in the network to reduce the risk of communication issues. The content is regularly recrawled from the original website if it is still available. If a new version of the AU is available, the previous version is kept but only the most recent AUs will be checked for fixity. <br />
:<br />
: &#8212;From [https://plnwiki.lockss.org/index.php?title=LOCKSS_Technical_Manual LOCKSS Technical Manual: Basic Private LOCKSS Network infrastructure] (Plnwiki)<br />
<br />
=== From [[Getting Started with LOCKSS, Midge Coates, 2013-06|"Getting Started with LOCKSS" (June 2016)]] ===<br />
<br />
: Each [[Archival Unit]] (AU) must be Web-accessible so the [[LOCKSS daemon]] can get at it. That means that it needs to be put on a Web-accessible server computer. If you have a server, but it isn't Web-accessible or access to it is blocked (at the firewall, for example), the LOCKSS crawl won't work. You can use a firewall to protect your collection, but make sure the LOCKSS daemon has access. Check with your IT support person or system administrator to confirm that this is the case.<br />
:<br />
: [...]<br />
:<br />
: The next major part of getting collections ingested is preparing each collection’s [[manifest page]]. The manifest page is a basic HTML document that contains at least two things: the permission statement that gives the LOCKSS daemon the right to harvest the collection (at the foot of the document) and the base URL of the AU for the collection (or for a fraction of the collection).<br />
:<br />
: Here is an example of a fairly simple manifest page for a fairly simple collection (Auburn University’s Alabama Postcards images collection). Feel free to use it as a template. <br />
:<br />
: [[Image:ExampleAlabamaPostcardsCollectionManifest.png]]<br />
:<br />
: The base URL isn't visible if you've opened the manifest in a browser window, because it's located inside the HTML <a href> tag. To see this tag, open the HTML file in DreamWeaver or Notepad++ . If you're opening it in a browser window, go to View => Source to see the tags.<br />
:<br />
: I usually put all the URLs for the sub-folders in my AU, but the LOCKSS folks tell me it isn't really necessary, as the daemon will go to all the folders inside the original unless you tell it not to. So you could leave out everything except the base URL. NOTE: These URLS are '''not''' the CONTENTdm collection URLs; they're the URLs for the collections (or AUs) on your Web server.<br />
:<br />
: The manifest page should be kept in the same folder as the AU so that the LOCKSS daemon can find it easily. If you break a collection up into digestible chunks (50 GB or so), you should have a separate manifest file for each AU chunk, with each AU chunk in a separate folder, along with its own manifest page. I also like to do a metadata export from the relevant CONTENTdm collection as a tab-delimited text file and put a copy of that in each folder also, in case the CONTENTdm collection needs to be reconstituted after a disaster.<br />
:<br />
: So, if you have one AU for a collection, you need one manifest with the appropriate base URL. If you have two AUs, you need two manifests, one in each AU folder, each with the appropriate base URL for that particular AU. Here is a screenshot of the folder directory for the Alabama Postcards AU, so you can see what is in the folder. The manifest has a little box around it, so you can't miss it.<br />
:<br />
: [[Image:ExampleAlabamaPostcardsCollectionDirectoryListing.png]]<br />
:<br />
:<br />
: &#8212;From "[[Getting Started with LOCKSS, Midge Coates, 2013-06|Getting Started with LOCKSS]]" (June 2016) by Midge Coates</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Q:_Does_ADPNet_provide_a_way_to_add_or_update_content_to_an_AU_after_it_has_been_preserved_in_the_network%3F&diff=2021Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?2022-04-12T14:28:43Z<p>CJohnson: </p>
<hr />
<div>'''Q.''' ''I'm [[HOWTO: Package files for staging on the Drop Server|packaging content that I want preserved]] into an [[Archival Unit]] (AU) for preservation in ADPNet. This content comes from a growing collection that will continue to have new items added. Is there a way to add new or updated content to the AU after it has already been added and crawled by other hosts?''<br />
<br />
'''A.''' This is possible for the LOCKSS software to do, but we strongly discourage organizing [[AU]]s in this way if it possible to avoid doing so. Wherever possible, we encourage adopting packaging schemes that avoid or minimize the use of the capability.<br />
<br />
LOCKSS ''does'' support versioning of content in AUs, so it is always possible to re-stage an altered version of an AU and replace the old version of the AU with an updated version that adds, removes, or modifies the content of files.<br />
<br />
If it turns out you '''do''' need to add to or update contents of an AU, then here’s what you’d do, under the current drop server configuration:<br />
# Prepare your AU with the new or modified contents and simply upload to the same directory location on the drop server. The LOCKSS content harvesters treat content as an update to an existing AUs '''if and only if''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
# A collection '''DOES NOT''' need to have been stored continuously on the drop server in order to update it. If you previously un-staged the content in the AU that you want to update, and had someone on TPC mark the AU as "down," then all you need to do is re-upload the AU (all the content, not just the new/modified content), put it in a directory with the same name as last time, and then have someone on the TPC mark it as "up."<br />
# In general, after an AU has been fully crawled by the LOCKSS network, when the content is unstaged from the drop server, the top-level directory it was in should be retained, with something like a small README file to indicate that the directory is a location where an AU used to be stored.<br />
# If you have followed [[HOWTO: Package files for staging on the Drop Server|recommended packaging guidelines for the AU content]], then any changes to the content of the AU (adding a file, removing a file, or modifying any of the files in the set), the existing BagIt manifest will no longer validate against the modified content. You will need to re-do the BagIt manifest and checksums to reflect the new contents.<br />
<br />
All that said, in general, I tend to '''strongly recommend''' organizing AUs so that their contents will '''not''' have to change over time, if it is at all possible to do so.<br />
<br />
# This is partly because the design of LOCKSS makes it easier to maintain consensus if AUs don’t change, and partly because other components that we use in our guidelines (e.g. BagIt) work best when they are applied to immutable units of data rather than to changing datasets.<br />
# The easiest way to do this with large, constantly growing collections is usually to adopt a packaging scheme so that the items will be arranged in series, and then subdivided into consistent chunks, either by number of items or by date of creation.<br />
## '''EXAMPLE:''' ADAH Digitization Masters are arranged by their internal unique identifier numbers and then divvied up into blocks of 500 items each, with breath-taking and poetic names like '''Digitization-Masters-Q-numbers-Master-Q0000105001_Q0000105500m'''<br />
## '''EXAMPLE:''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is easier to organize it into blocks based on a fixed date (day or month of production) than it is to work with blocks based on a fixed count of items. AUs have names like '''repository-2020-11''', etc.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Q:_Does_ADPNet_provide_a_way_to_add_or_update_content_to_an_AU_after_it_has_been_preserved_in_the_network%3F&diff=2020Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?2022-04-12T14:26:59Z<p>CJohnson: </p>
<hr />
<div>'''Q.''' ''I'm [[HOWTO: Package files for staging on the Drop Server|packaging content that I want preserved]] into an [[Archival Unit]] (AU) for preservation in ADPNet. This content comes from a growing collection that will continue to have new items added. Is there a way to add new or updated content to the AU after it has already been added and crawled by other hosts?''<br />
<br />
'''A.''' This is possible for the LOCKSS software to do, but we strongly discourage organizing AUs in this way if it possible to avoid doing so. Wherever possible, we encourage adopting packaging schemes that avoid or minimize the use of the capability.<br />
<br />
LOCKSS ''does'' support versioning of content in AUs, so it is always possible to re-stage an altered version of an AU and replace the old version of the AU with an updated version that adds, removes, or modifies the content of files.<br />
<br />
If it turns out you '''do''' need to add to or update contents of an AU, then here’s what you’d do, under the current drop server configuration:<br />
# Prepare your AU with the new or modified contents and simply upload to the same directory location on the drop server. The LOCKSS content harvesters treat content as an update to an existing AUs '''if and only if''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
# A collection '''DOES NOT''' need to have been stored continuously on the drop server in order to update it. If you previously un-staged the content in the AU that you want to update, and had someone on TPC mark the AU as "down," then all you need to do is re-upload the AU (all the content, not just the new/modified content), put it in a directory with the same name as last time, and then have someone on the TPC mark it as "up."<br />
# In general, after an AU has been fully crawled by the LOCKSS network, when the content is unstaged from the drop server, the top-level directory it was in should be retained, with something like a small README file to indicate that the directory is a location where an AU used to be stored.<br />
# If you have followed [[HOWTO: Package files for staging on the Drop Server|recommended packaging guidelines for the AU content]], then any changes to the content of the AU (adding a file, removing a file, or modifying any of the files in the set), the existing BagIt manifest will no longer validate against the modified content. You will need to re-do the BagIt manifest and checksums to reflect the new contents.<br />
<br />
All that said, in general, I tend to '''strongly recommend''' organizing AUs so that their contents will '''not''' have to change over time, if it is at all possible to do so.<br />
<br />
# This is partly because the design of LOCKSS makes it easier to maintain consensus if AUs don’t change, and partly because other components that we use in our guidelines (e.g. BagIt) work best when they are applied to immutable units of data rather than to changing datasets.<br />
# The easiest way to do this with large, constantly growing collections is usually to adopt a packaging scheme so that the items will be arranged in series, and then subdivided into consistent chunks, either by number of items or by date of creation.<br />
## '''EXAMPLE:''' ADAH Digitization Masters are arranged by their internal unique identifier numbers and then divvied up into blocks of 500 items each, with breath-taking and poetic names like '''Digitization-Masters-Q-numbers-Master-Q0000105001_Q0000105500m'''<br />
## '''EXAMPLE:''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is easier to organize it into blocks based on a fixed date (day or month of production) than it is to work with blocks based on a fixed count of items. AUs have names like '''repository-2020-11''', etc.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Archival_Unit&diff=2019Archival Unit2022-04-12T14:23:38Z<p>CJohnson: cross-link to some HOWTO and Q/A pages</p>
<hr />
<div>An [[Archival Unit]] (AU) is the basic unit of preservation on the ADPNet LOCKSS network. Each AU has a [[Manifest Page]] and a collection of digital assets that the manifest page links to.<br />
<br />
== Network Members ==<br />
* [[HOWTO: Package files for staging on the Drop Server]]<br />
* [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]<br />
<br />
== Preservation Node Managers ==<br />
<br />
* [[HOWTO: Add a new AU to your node for preservation]]<br />
<br />
== From [https://plnwiki.lockss.org/index.php?title=LOCKSS_Technical_Manual LOCKSS Technical Manual] ==<br />
<br />
: LOCKSS caches: The LOCKSS caches are the nodes in the network in which digital objects will be preserved and monitored as archival units. One archival unit is typically a one-year collection (size: 1GB to 20GB). The size of an AU results of a trade-off between the processing overhead required by large AUs and the guarantee that all AUs integrity can be regularly checked in the case of a multitude of small AUs. The daemon is a java application which collects digital objects through http requests from the original website, store them inside the cache as an Archival Unit, computes an SHA-1 checksum and regularly monitors their integrity by comparing the preserved content with the other caches in the network with a specific voting protocol (LCAP). The AUs are collected at various moments in time by the different nodes in the network to reduce the risk of communication issues. The content is regularly recrawled from the original website if it is still available. If a new version of the AU is available, the previous version is kept but only the most recent AUs will be checked for fixity. <br />
:<br />
: &#8212;From [https://plnwiki.lockss.org/index.php?title=LOCKSS_Technical_Manual LOCKSS Technical Manual: Basic Private LOCKSS Network infrastructure] (Plnwiki)<br />
<br />
== From [[Getting Started with LOCKSS, Midge Coates, 2013-06|"Getting Started with LOCKSS" (June 2016)]] ==<br />
<br />
: Each [[Archival Unit]] (AU) must be Web-accessible so the [[LOCKSS daemon]] can get at it. That means that it needs to be put on a Web-accessible server computer. If you have a server, but it isn't Web-accessible or access to it is blocked (at the firewall, for example), the LOCKSS crawl won't work. You can use a firewall to protect your collection, but make sure the LOCKSS daemon has access. Check with your IT support person or system administrator to confirm that this is the case.<br />
:<br />
: [...]<br />
:<br />
: The next major part of getting collections ingested is preparing each collection’s [[manifest page]]. The manifest page is a basic HTML document that contains at least two things: the permission statement that gives the LOCKSS daemon the right to harvest the collection (at the foot of the document) and the base URL of the AU for the collection (or for a fraction of the collection).<br />
:<br />
: Here is an example of a fairly simple manifest page for a fairly simple collection (Auburn University’s Alabama Postcards images collection). Feel free to use it as a template. <br />
:<br />
: [[Image:ExampleAlabamaPostcardsCollectionManifest.png]]<br />
:<br />
: The base URL isn't visible if you've opened the manifest in a browser window, because it's located inside the HTML <a href> tag. To see this tag, open the HTML file in DreamWeaver or Notepad++ . If you're opening it in a browser window, go to View => Source to see the tags.<br />
:<br />
: I usually put all the URLs for the sub-folders in my AU, but the LOCKSS folks tell me it isn't really necessary, as the daemon will go to all the folders inside the original unless you tell it not to. So you could leave out everything except the base URL. NOTE: These URLS are '''not''' the CONTENTdm collection URLs; they're the URLs for the collections (or AUs) on your Web server.<br />
:<br />
: The manifest page should be kept in the same folder as the AU so that the LOCKSS daemon can find it easily. If you break a collection up into digestible chunks (50 GB or so), you should have a separate manifest file for each AU chunk, with each AU chunk in a separate folder, along with its own manifest page. I also like to do a metadata export from the relevant CONTENTdm collection as a tab-delimited text file and put a copy of that in each folder also, in case the CONTENTdm collection needs to be reconstituted after a disaster.<br />
:<br />
: So, if you have one AU for a collection, you need one manifest with the appropriate base URL. If you have two AUs, you need two manifests, one in each AU folder, each with the appropriate base URL for that particular AU. Here is a screenshot of the folder directory for the Alabama Postcards AU, so you can see what is in the folder. The manifest has a little box around it, so you can't miss it.<br />
:<br />
: [[Image:ExampleAlabamaPostcardsCollectionDirectoryListing.png]]<br />
:<br />
:<br />
: &#8212;From "[[Getting Started with LOCKSS, Midge Coates, 2013-06|Getting Started with LOCKSS]]" (June 2016) by Midge Coates</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Q:_Does_ADPNet_provide_a_way_to_add_or_update_content_to_an_AU_after_it_has_been_preserved_in_the_network%3F&diff=2018Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?2022-04-08T17:55:10Z<p>CJohnson: </p>
<hr />
<div>'''Q.''' ''I'm [[HOWTO: Package files for staging on the Drop Server|packaging content that I want preserved into an Archival Unit]] (AU) for preservation in ADPNet. This content comes from a growing collection that will continue to have new items added. Is there a way to add new or updated content to the AU after it has already been added and crawled by other hosts?''<br />
<br />
'''A.''' This is possible for the LOCKSS software to do, but we strongly discourage organizing AUs in this way if it possible to avoid doing so. Wherever possible, we encourage adopting packaging schemes that avoid or minimize the use of the capability.<br />
<br />
LOCKSS ''does'' support versioning of content in AUs, so it is always possible to re-stage an altered version of an AU and replace the old version of the AU with an updated version that adds, removes, or modifies the content of files.<br />
<br />
If it turns out you '''do''' need to add to or update contents of an AU, then here’s what you’d do, under the current drop server configuration:<br />
# Prepare your AU with the new or modified contents and simply upload to the same directory location on the drop server. The LOCKSS content harvesters treat content as an update to an existing AUs '''if and only if''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
# A collection '''DOES NOT''' need to have been stored continuously on the drop server in order to update it. If you previously un-staged the content in the AU that you want to update, and had someone on TPC mark the AU as "down," then all you need to do is re-upload the AU (all the content, not just the new/modified content), put it in a directory with the same name as last time, and then have someone on the TPC mark it as "up."<br />
# In general, after an AU has been fully crawled by the LOCKSS network, when the content is unstaged from the drop server, the top-level directory it was in should be retained, with something like a small README file to indicate that the directory is a location where an AU used to be stored.<br />
# If you have followed [[HOWTO: Package files for staging on the Drop Server|recommended packaging guidelines for the AU content]], then any changes to the content of the AU (adding a file, removing a file, or modifying any of the files in the set), the existing BagIt manifest will no longer validate against the modified content. You will need to re-do the BagIt manifest and checksums to reflect the new contents.<br />
<br />
All that said, in general, I tend to '''strongly recommend''' organizing AUs so that their contents will '''not''' have to change over time, if it is at all possible to do so.<br />
<br />
# This is partly because the design of LOCKSS makes it easier to maintain consensus if AUs don’t change, and partly because other components that we use in our guidelines (e.g. BagIt) work best when they are applied to immutable units of data rather than to changing datasets.<br />
# The easiest way to do this with large, constantly growing collections is usually to adopt a packaging scheme so that the items will be arranged in series, and then subdivided into consistent chunks, either by number of items or by date of creation.<br />
## '''EXAMPLE:''' ADAH Digitization Masters are arranged by their internal unique identifier numbers and then divvied up into blocks of 500 items each, with breath-taking and poetic names like '''Digitization-Masters-Q-numbers-Master-Q0000105001_Q0000105500m'''<br />
## '''EXAMPLE:''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is easier to organize it into blocks based on a fixed date (day or month of production) than it is to work with blocks based on a fixed count of items. AUs have names like '''repository-2020-11''', etc.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Q:_Does_ADPNet_provide_a_way_to_add_or_update_content_to_an_AU_after_it_has_been_preserved_in_the_network%3F&diff=2017Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?2022-04-08T17:54:49Z<p>CJohnson: </p>
<hr />
<div>'''Q.''' ''I'm [[HOWTO: Package files for staging on the Drop Server|packaging content that I want preserved into an Archival Unit]] (AU) for preservation in ADPNet. This content comes from a growing collection that will continue to have new items added. Is there a way to add new or updated content to the AU after it has already been added and crawled by other hosts?''<br />
<br />
'''A.''' This is possible for the LOCKSS software to do, but we strongly discourage organizing AUs in this way if it possible to avoid doing so. Wherever possible, we encourage adopting packaging schemes that avoid or minimize the use of the capability.<br />
<br />
LOCKSS ''does'' support versioning of content in AUs, so it is always possible to re-stage an altered version of an AU and replace the old version of the AU with an updated version that adds, removes, or modifies the content of files.<br />
<br />
# If it turns out you '''do''' need to add to or update contents of an AU, then here’s what you’d do, under the current drop server configuration:<br />
## Prepare your AU with the new or modified contents and simply upload to the same directory location on the drop server. The LOCKSS content harvesters treat content as an update to an existing AUs '''if and only if''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
## A collection '''DOES NOT''' need to have been stored continuously on the drop server in order to update it. If you previously un-staged the content in the AU that you want to update, and had someone on TPC mark the AU as "down," then all you need to do is re-upload the AU (all the content, not just the new/modified content), put it in a directory with the same name as last time, and then have someone on the TPC mark it as "up."<br />
## In general, after an AU has been fully crawled by the LOCKSS network, when the content is unstaged from the drop server, the top-level directory it was in should be retained, with something like a small README file to indicate that the directory is a location where an AU used to be stored.<br />
## If you have followed [[HOWTO: Package files for staging on the Drop Server|recommended packaging guidelines for the AU content]], then any changes to the content of the AU (adding a file, removing a file, or modifying any of the files in the set), the existing BagIt manifest will no longer validate against the modified content. You will need to re-do the BagIt manifest and checksums to reflect the new contents.<br />
<br />
All that said, in general, I tend to '''strongly recommend''' organizing AUs so that their contents will '''not''' have to change over time, if it is at all possible to do so.<br />
# This is partly because the design of LOCKSS makes it easier to maintain consensus if AUs don’t change, and partly because other components that we use in our guidelines (e.g. BagIt) work best when they are applied to immutable units of data rather than to changing datasets.<br />
# The easiest way to do this with large, constantly growing collections is usually to adopt a packaging scheme so that the items will be arranged in series, and then subdivided into consistent chunks, either by number of items or by date of creation.<br />
## '''EXAMPLE:''' ADAH Digitization Masters are arranged by their internal unique identifier numbers and then divvied up into blocks of 500 items each, with breath-taking and poetic names like '''Digitization-Masters-Q-numbers-Master-Q0000105001_Q0000105500m'''<br />
## '''EXAMPLE:''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is easier to organize it into blocks based on a fixed date (day or month of production) than it is to work with blocks based on a fixed count of items. AUs have names like '''repository-2020-11''', etc.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Q:_Does_ADPNet_provide_a_way_to_add_or_update_content_to_an_AU_after_it_has_been_preserved_in_the_network%3F&diff=2016Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?2022-04-08T17:54:19Z<p>CJohnson: Created page with "'''Q.''' ''I'm packaging content that I want preserved into an Archival Unit (AU) for preservation in ADPNet. This cont..."</p>
<hr />
<div>'''Q.''' ''I'm [[HOWTO: Package files for staging on the Drop Server|packaging content that I want preserved into an Archival Unit]] (AU) for preservation in ADPNet. This content comes from a growing collection that will continue to have new items added. Is there a way to add new or updated content to the AU after it has already been added and crawled by other hosts?''<br />
<br />
'''A.''' This is possible for the LOCKSS software to do, but we strongly discourage organizing AUs in this way if it possible to avoid doing so. Wherever possible, we encourage adopting packaging schemes that avoid or minimize the use of the capability.<br />
<br />
# LOCKSS does support versioning of content in AUs, so it is always possible to re-stage an altered version of an AU and replace the old version of the AU with an updated version that adds, removes, or modifies the content of files.<br />
# If it turns out you '''do''' need to add to or update contents of an AU, then here’s what you’d do, under the current drop server configuration:<br />
## Prepare your AU with the new or modified contents and simply upload to the same directory location on the drop server. The LOCKSS content harvesters treat content as an update to an existing AUs '''if and only if''' the content is placed under '''exactly the same directory name''' as the AU when originally ingested.<br />
## A collection '''DOES NOT''' need to have been stored continuously on the drop server in order to update it. If you previously un-staged the content in the AU that you want to update, and had someone on TPC mark the AU as "down," then all you need to do is re-upload the AU (all the content, not just the new/modified content), put it in a directory with the same name as last time, and then have someone on the TPC mark it as "up."<br />
## In general, after an AU has been fully crawled by the LOCKSS network, when the content is unstaged from the drop server, the top-level directory it was in should be retained, with something like a small README file to indicate that the directory is a location where an AU used to be stored.<br />
## If you have followed [[HOWTO: Package files for staging on the Drop Server|recommended packaging guidelines for the AU content]], then any changes to the content of the AU (adding a file, removing a file, or modifying any of the files in the set), the existing BagIt manifest will no longer validate against the modified content. You will need to re-do the BagIt manifest and checksums to reflect the new contents.<br />
# All that said, in general, I tend to '''strongly recommend''' organizing AUs so that their contents will '''not''' have to change over time, if it is at all possible to do so.<br />
## This is partly because the design of LOCKSS makes it easier to maintain consensus if AUs don’t change, and partly because other components that we use in our guidelines (e.g. BagIt) work best when they are applied to immutable units of data rather than to changing datasets.<br />
## The easiest way to do this with large, constantly growing collections is usually to adopt a packaging scheme so that the items will be arranged in series, and then subdivided into consistent chunks, either by number of items or by date of creation.<br />
### '''EXAMPLE:''' ADAH Digitization Masters are arranged by their internal unique identifier numbers and then divvied up into blocks of 500 items each, with breath-taking and poetic names like '''Digitization-Masters-Q-numbers-Master-Q0000105001_Q0000105500m'''<br />
### '''EXAMPLE:''' Tuskegee's digitized content has a production schedule that is much more up and down, so it is easier to organize it into blocks based on a fixed date (day or month of production) than it is to work with blocks based on a fixed count of items. AUs have names like '''repository-2020-11''', etc.</div>CJohnsonhttp://www.adpn.org/w/index.php?title=Main_Page&diff=2015Main Page2022-04-08T17:39:38Z<p>CJohnson: /* HOWTO */</p>
<hr />
<div>== ALABAMA DIGITAL PRESERVATION NETWORK ==<br />
<br />
* [[Current events|ADPNet Current Events]]<br />
<br />
* [http://www.adpn.org/docs/pdf/ADPNet_Governance_Policy.pdf ADPNet Governance Policy]<br />
* [http://www.adpn.org/docs/pdf/ADPNet_Membership_Model_20111001.pdf ADPNet Membership Model]<br />
* [https://gitlab.com/adpnet ADPNet Gitlab Repository]<br />
<br />
The Alabama Digital Preservation Network (ADPNet) is a distributed digital preservation network, maintained by a number of Alabama libraries, archives, colleges and universities [[The Project|since 2008]]. Member institutions pool their electronic resources through a [[LOCKSS Basics|LOCKSS]]-based network of [[preservation nodes]] in order to provide a long-term storage repository for digital assets. The network can support repositories of different types and sizes as well as different kinds of institutions, and it provides secure, geographically distributed, low-cost, and member-run storage for the long-term preservation of digital assets. ADPNet is governed by the [[Member (ADPNet)|member institutions]] through the network's [[Steering Committee]]; the network is financially supported by [[Member (ADPNet)|member institutions]], and sponsored by the [https://ache.edu/NAAL.aspx Network of Alabama Academic Libraries] (NAAL).<br />
<br />
We welcome new [[Member (ADPNet)|members]] from Alabama and the U.S. Southeast. [http://adpn.org/form.html Contact Us with a Membership Request] if you are interested in joining the digital preservation network.<br />
<br />
=== Committees and Workgroups ===<br />
<br />
* The '''[[Steering Committee]]''' sets general ADPNet policy, reviews and approves requests to join ADPNet, and reviews and approves requests to increase ADPNet storage capacity. Each [[Member (ADPNet)|Member]] of ADPNet appoints voting representatives to the Steering Committee according to [[membership categories]], for a term of 1 year.<br />
<br />
* The '''[[Technical Policy Committee]]''' periodically reviews the ADPNet capacity and technical specifications and prepares recommendations on issues having to do with hardware and software. It consists of members with technical expertise representing the [[ADPNet Host]] member organizations. Chairs of the [[Steering Committee]] serve as ''ex officio'' members of the committee, and appoint regular committee members from the staff of [[ADPNet Host|Host Members]].<br />
<br />
=== Q&A: Network Members ===<br />
* [[Q: Does ADPNet provide a way to add or update content to an AU after it has been preserved in the network?]]<br />
<br />
=== HOWTO: Network Members ===<br />
* [[HOWTO: Request a staging area on the Drop Server]]<br />
* [[HOWTO: Package files for staging on the Drop Server]]<br />
<br />
==== HOWTO: Preservation Node Managers ====<br />
* [[HOWTO: Add a new AU to your node for preservation]]<br />
* [[HOWTO: Check the version of Java running on your preservation node]]<br />
<br />
== Network History ==<br />
* [[The Project|Project Prospectus]] (February 2008)<br />
<br />
== LOCKSS Resources ==<br />
* [[Getting_Started|Getting Started]] with LOCKSS<br />
** [http://www.lockss.org/support/build-a-lockss-box/ Installation Instructions]<br />
** [http://sourceforge.net/projects/lockss/files/Plugin%20Tool/ Plugin Tool Download] (0.12.1 latest from SourceForge)<br />
** [http://web.archive.org/web/20120201154913/http://www.lockss.org/lockss/Plugin_Tool_Tutorial LOCKSS Plugin Tool Tutorial] (Way Back Machine IA)<br />
** [http://www.adpn.org/docs/pdf/ADPNAnnotation.pdf Annotated ADPNet LOCKSS Plugin]<br />
* Learn more about [[LOCKSS_Software |LOCKSS Software]]<br />
<br />
<br />
<br />
__NOTOC__</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_preservation&diff=1975HOWTO: Add a new AU to your node for preservation2021-08-11T16:13:42Z<p>CJohnson: </p>
<hr />
<div>:''This HOWTO document is for '''[[Preservation Node]] Managers''' who have been asked to add an [[Archival Unit]] (AU) to their LOCKSS node for preservation.''<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is ready for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]].<br />
<br />
Here's how you do that:<br />
<br />
== 1. Log in to your LOCKSS Administrative Interface ==<br />
<br />
[[File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png]]<br />
<br />
== 2. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png]]<br />
<br />
[[File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png]]<br />
<br />
== 3. Find the new AU under '''Journal Configuration''' > '''Add AUs''' ==<br />
<br />
[[File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png]]<br />
<br />
[[File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png]]<br />
<br />
== 4. Select the institution using "Select AUs" ==<br />
== 5. Select the new AU and mash "Add Selected AUs" ==</div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png&diff=1974File:Screenshot-20210811-110933-LOCKSS-Journal-Configuration-Selected-Add-AUs.png2021-08-11T16:12:58Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png&diff=1973File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Selected-Journal-Configuration.png2021-08-11T16:08:59Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png&diff=1972File:Screenshot-20210811-104929-LOCKSS-Debug-Panel-Selected-Reload-Config.png2021-08-11T16:04:25Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png&diff=1971File:Screenshot-20210811-104745-LOCKSS-LOCKSS-Administration-Daemon-Status-Selected.png2021-08-11T16:04:13Z<p>CJohnson: </p>
<hr />
<div></div>CJohnsonhttp://www.adpn.org/w/index.php?title=File:Screenshot_2021-08-11_at_10-47-45_LOCKSS_LOCKSS_Administration.png&diff=1970File:Screenshot 2021-08-11 at 10-47-45 LOCKSS LOCKSS Administration.png2021-08-11T15:59:35Z<p>CJohnson: Screenshot, LOCKSS Administrative Panel front page</p>
<hr />
<div>== Summary ==<br />
Screenshot, LOCKSS Administrative Panel front page</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_preservation&diff=1969HOWTO: Add a new AU to your node for preservation2021-08-11T15:56:03Z<p>CJohnson: use ordered list</p>
<hr />
<div>:''This HOWTO document is for '''[[Preservation Node]] Managers''' who have been asked to add an [[Archival Unit]] (AU) to their LOCKSS node for preservation.''<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is ready for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]].<br />
<br />
Here's how you do that:<br />
<br />
# Log in to your LOCKSS Administrative Interface<br />
# Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config'''<br />
# Find the new AU under '''Journal Configuration''' > '''Add AUs'''<br />
# Select the institution using "Select AUs"<br />
# Select the new AU and mash "Add Selected AUs"</div>CJohnsonhttp://www.adpn.org/w/index.php?title=HOWTO:_Add_a_new_AU_to_your_node_for_preservation&diff=1968HOWTO: Add a new AU to your node for preservation2021-08-11T15:55:44Z<p>CJohnson: quick start; add screen shots later when I have a good example</p>
<hr />
<div>:''This HOWTO document is for '''[[Preservation Node]] Managers''' who have been asked to add an [[Archival Unit]] (AU) to their LOCKSS node for preservation.''<br />
<br />
So, you have been informed that a new [[Archival Unit]] (AU) is ready for preservation in ADPNet, and you have been asked to add the AU to your LOCKSS [[Preservation Node]].<br />
<br />
Here's how you do that:<br />
<br />
1. Log in to your LOCKSS Administrative Interface<br />
2. Refresh your AU titles feed using '''Debug Panel''' > '''Reload Config'''<br />
3. Find the new AU under '''Journal Configuration''' > '''Add AUs'''<br />
4. Select the institution using "Select AUs"<br />
5. Select the new AU and mash "Add Selected AUs"</div>CJohnson