The robots.txt file for any site will tell search engines what directories or files to ignore on a website. Usually this is used to hide directories that are still under construction, or that have been moved or changed.
Check out the robots.txt file for the white house website to see a list of things that the White House has either removed from the site, or no longer wants the public to access.