This general idea (though not the specifics) is how it works. We have automated systems that determine on the fly (in memory) whether something is a bot or not and then mark it as such. The most prevalent bot is actually Googlebot, which is extremely easy to detect in nginx.

As for the AWS question, we use encryption by default, which encrypts HTTP headers and those the search query. Actually the end server isn't all that matters. If you don't use HTTPS then actually lots of servers between you and us could intercept your search term. Just do a trace route between you and any Internet site. However, with encryption these headers are encrypted when they travel across the Internet.
posted by yegg Staff4 years and 1 month ago Link

The search terms are not part of the headers but part of the URL in a GET request. For a POST request those terms are part of the body.

In any event, even before DDG has a chance to encrypt any data Amazon has already logged my search query and IP address.
posted by <hidden> • 4 years and 1 month ago Link
This a common misconception. All headers (including the full URL and search terms embedded in it) are encrypted on a GET request:
posted by yegg Staff4 years and 1 month ago Link
This is a good time to mention:
posted by Jlg Community Leader4 years and 1 month ago Link