How to Get ChatGPT and AI Search to Read Your Website: llms.txt, Structured Data, and Crawler Settings

More and more people are asking ChatGPT things like "What's a good 〇〇?", and AI assistants are becoming a gateway for choosing products and services. AI doesn't answer from memory alone; with each question, it searches the web, reads pages, and builds its answer. In other words, whether AI cites you depends on whether AI crawlers can read your pages and whether the facts it needs are written out as text when it reads them. This article walks through the settings required, step by step.

1. Check that you aren't blocking AI crawlers

The crawlers used by the major AI services are listed below. If you Disallow them in robots.txt, you are unlikely to appear in AI answers at all.

Note that some CDNs, such as Cloudflare, may have a "Block AI crawlers" setting turned on by default for new sites. Check the bot settings in your dashboard. Also, if your site isn't registered with Bing, it will be less likely to appear in ChatGPT search results. Bing Webmaster Tools can import your Google Search Console settings, so registration takes only a few minutes.

2. Declare Your Preferences in robots.txt (Content Signals)

Since 2025, a format called Content Signals has come into use, allowing you to state preferences in robots.txt such as "may be used for search," "may be used in AI answers," and "may not be used for training." Here is an example:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://example.co.jp/sitemap.xml

ai-input=yes means "may be used in AI answers"; set it to yes if you want to be cited. Use for training (ai-train) can be decided separately.

3. Add an llms.txt File

/llms.txt is a guide file for AI that provides an overview of your site and a list of its key pages, written in Markdown. Much like a sitemap for humans, it lets AI quickly understand your entire site.

# 会社名 — 一言でサービスの説明

> 2〜3文の説明。誰向けに、何を、いくらで。

## 主要ページ
- [料金](https://example.co.jp/pricing): プランと価格
- [よくある質問](https://example.co.jp/faq): 導入前の疑問
- [会社概要](https://example.co.jp/company): 所在地、代表者、沿革

For sites served by Sotomuke, an llms.txt file listing translated pages in all languages is generated automatically and is also referenced in the HTTP headers of each page.

4. Write Facts as Text

What AI cites is the text written in the body of a page. If your price list exists only as an image, your business hours appear only in a banner, or your features are described only in slides, AI will not pick them up. Write the following items as body text:

• What you offer, to whom, and at what price (state prices as numbers)
• Location, business hours, and contact information
• What sets you apart (facts used in comparisons, such as "supports XX" or "no XX required")
• Frequently asked questions and answers (use the question itself as the heading)

5. Add Structured Data (JSON-LD)

This is a way to attach a machine-readable version of your page content. For company information, use Organization; for stores, use LocalBusiness; for products and services, use Product or SoftwareApplication; for questions and answers, use FAQPage; and for articles, use Article . Structuring prices, business hours, and location lets both search engines and AI pick them up accurately.

<script type="application/ld+json">
{ "@context": "https://schema.org", "@type": "LocalBusiness",
  "name": "店名", "url": "https://example.co.jp/",
  "address": { "@type": "PostalAddress", "addressLocality": "京都市", "addressCountry": "JP" },
  "openingHours": "Mo-Su 11:00-22:00", "priceRange": "¥¥" }
</script>

6. Provide Multilingual Pages

AI searches in the language the question was asked in and prioritizes pages in that language. If someone asks "best ryokan in Kanazawa" in English and you have no English page, you are less likely to be cited. Set up each language with its own URL and have the server return translated HTML (with widgets that swap text in the browser, AI crawlers still see the page in Japanese). Linking language versions with hreflang also ensures search engines handle them properly.

7. Serve Markdown as Well (Optional)

Some AI agents request Markdown, which is lighter than HTML, using Accept: text/markdown . Supporting this reduces noise when AI reads your pages. Pages served by Sotomuke automatically return Markdown in response to this request.

How to Check

• Ask ChatGPT questions like "Tell me about [your company name]" or "What [product/service] do you recommend in [city]?" and see whether your website is cited. The source links will be displayed.
• On Bing, search for site:example.co.jp and check whether your pages are indexed.
• Use Search Console and Bing Webmaster Tools to check indexing status and crawl errors.

Translated pages published with Sotomuke include language-specific URLs, hreflang, sitemaps, llms.txt, Content Signals, and Markdown delivery from the start. For your Japanese site, add these settings to your site by following the steps above.