Search Appliance SBE
Syntax: one or more URLs, one per line
Here you may specify URLs for individual pages to include in the index. These pages are fetched and stored in the database like others but the hyperlinks on them are not followed during a walk.
Pages removed from the list will not be removed from the database
until the next New walk.
Note that since this setting is intended for "one-off" individual URLs, robots.txt will not be fetched for these pages.