Robots.txt

Definition of robots.txt

user-agent:*(where*=all bots)
disallow:/about us (only one page)
disallow:/ full website block)
allow:

definition:its is a set of programs(crawler,spider) which are used to disallow or allow a webpage for search engine.