Although Windows Server 2012 R2/2016 comes with an IFilter for .rtf-Files, SharePoint does not crawl/index .rtf files out of the box.
The following article lists .rtf as a crawled filename, but it also states, that .rtf is not in the Managed File Types page by default: https://technet.microsoft.com/en-us/library/jj219530.aspx
What happens when you add .rtf to the Managed File Type in the SharePoint Enterprise Search Service Application and Restart OSearch16? The file gets crawled, but with an error saying the file is “in a wrong format”.
Resolution: you need to tell SharePoint to use the rtffilt.dll (which is the .rtf IFilter) for .rtf-Files by adding two Registry Keys:
new-item -Path “HKLM:\software\microsoft\Shared Tools\Web Server Extensions\16.0\search\setup\ContentIndexCommon\Filters\clsid{e2403e98-663b-4df6-b234-687789db8560}"
Set-ItemProperty -Path “HKLM:\software\microsoft\Shared Tools\Web Server Extensions\16.0\search\setup\ContentIndexCommon\Filters\clsid{e2403e98-663b-4df6-b234-687789db8560}" -Name "(default)" -type MultiString -Value "{e2403e98-663b-4df6-b234-687789db8560}"
new-item -Path “HKLM:\software\microsoft\Shared Tools\Web Server Extensions\16.0\search\setup\ContentIndexCommon\Filters\extension.rtf”
Set-ItemProperty -Path “HKLM:\software\microsoft\Shared Tools\Web Server Extensions\16.0\search\setup\ContentIndexCommon\Filters\extension.rtf” -Name "(default)" -type MultiString -Value "{e2403e98-663b-4df6-b234-687789db8560}"
After restarting OSearch16 (restart-service osearch16), .rtf files should be crawled and indexed in SharePoint 2016 just fine.
Markus