Robots Txt

How do I modify robots.txt?

How do I modify robots.txt?

You can modify the robots.txt file by extending the \Aero\Store\Pipelines\RobotsTxt pipeline.

For example, you can prevent search engines from crawling the admin login page:

\Aero\Store\Pipelines\RobotsTxt::extend(function (\Aero\Store\Collections\RobotsTxtCollection $content, \Closure $next) {
    $adminSlug = config('aero.admin.slug');

    $content->agent('*')->disallow("/{$adminSlug}");

    return $next($content);
});

Parameters

Parameter	Type	Description
`$content`	`\Aero\Store\Collections\RobotsTxtCollection`	The collection of rules for the robots.txt file.
`$next`	`\Closure`	The next step in the pipeline.

Methods

Method	Description
`agent(string $agent)`	Gets or creates a new rule collection for the specified user agent.
`removeAgent(string $agent)`	Removes a user agent entirely (including all of its defined rules).
`allow(string $value)`	Adds an `Allow: $value` rule.
`removeAllow(string $value)`	Removes an `Allow: $value` rule.
`disallow(string $value)`	Adds an `Disallow: $value` rule.
`removeDisallow(string $value)`	Removes an `Disallow: $value` rule.
`sitemap(string $value)`	Adds an `Sitemap: $value` rule.
`removeSitemap(string $value)`	Removes an `Sitemap: $value` rule.
`crawlDelay(string $value)`	Adds an `Crawl-delay: $value` rule.
`removeCrawlDelay(string $value)`	Removes an `Crawl-delay: $value` rule.
`requestRate(string $value)`	Adds an `Request-rate: $value` rule.
`removeRequestRate(string $value)`	Removes an `Request-rate: $value` rule.