Autocomplete with Django-Haystack and Elasticsearch with single letter querying.

Django's haystack provides autocomplete functionality. To do autocomplete effectively, the search backend(elasticsearch in this case) uses n-grams (essentially a small window passed over the string). Because this alters the way your data needs to be stored. We had two choices: NgramField and EdgeNgramField used as n-grams in search backend. The major drawback of the n-grams is that they take minimum of 3 letters in the search query. Still EdgeNgramField or NgramField fields won't produce consistent results(For Eg: . By customising the Haystack's built in elastcisearch engine backend we can achieve single letter query with Charfield itself.

In your search_indexes.py

from haystack import indexes
from myapp.models import Book

class BookIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    title = indexes.CharField(model_attr='title')
    pub_date = indexes.DateTimeField(model_attr='pub_date')

    def get_model(self):
        return Book

Create backends.py in your app. And this file will contain the actual customized elasticsearch search engine backend.

from django.conf import settings
from haystack.backends.elasticsearch_backend import ElasticsearchSearchBackend
from haystack.backends.elasticsearch_backend import ElasticsearchSearchEngine
from haystack.backends.elasticsearch_backend import ElasticsearchSearchQuery
from haystack.constants import VALID_FILTERS, FILTER_SEPARATOR, DEFAULT_ALIAS
from haystack.inputs import Clean, Exact, PythonData, Raw
from django.utils import six

class CustomElasticsearchSearchQuery(ElasticsearchSearchQuery):

    def __init__(self, using=DEFAULT_ALIAS):
        super(CustomElasticsearchSearchQuery, self).__init__(using=DEFAULT_ALIAS)

    def build_query_fragment(self, field, filter_type, value):
        from haystack import connections
        query_frag = ''

        if not hasattr(value, 'input_type_name'):
            # Handle when we've got a ``ValuesListQuerySet``...
            if hasattr(value, 'values_list'):
                value = list(value)

            if isinstance(value, six.string_types):
                # It's not an ``InputType``. Assume ``Clean``.
                value = Clean(value)
            else:
                value = PythonData(value)

        # Prepare the query using the InputType.
        prepared_value = value.prepare(self)

        if not isinstance(prepared_value, (set, list, tuple)):
            # Then convert whatever we get back to what pysolr wants if needed.
            prepared_value = self.backend._from_python(prepared_value)

        # 'content' is a special reserved word, much like 'pk' in
        # Django's ORM layer. It indicates 'no special field'.
        if field == 'content':
            index_fieldname = ''
        else:
            index_fieldname = u'%s:' % connections[self._using].get_unified_index().get_index_fieldname(field)

        filter_types = {
            'contains': u'*%s*',
            'startswith': u'%s*',
            'exact': u'%s',
            'gt': u'{%s TO *}',
            'gte': u'[%s TO *]',
            'lt': u'{* TO %s}',
            'lte': u'[* TO %s]',
        }

        if value.post_process is False:
            query_frag = prepared_value
        else:
            if filter_type in ['contains', 'startswith']:
                if value.input_type_name == 'exact':
                    query_frag = prepared_value
                else:
                    # Iterate over terms & incorportate the converted form of each into the query.
                    terms = []

                    if isinstance(prepared_value, six.string_types):
                        for possible_value in prepared_value.split(' '):
                            terms.append(filter_types[filter_type] % self.backend._from_python(possible_value))
                    else:
                        terms.append(filter_types[filter_type] % self.backend._from_python(prepared_value))

                    if len(terms) == 1:
                        query_frag = terms[0]
                    else:
                        query_frag = u"(%s)" % " AND ".join(terms)
            elif filter_type == 'in':
                in_options = []

                for possible_value in prepared_value:
                    in_options.append(u'"%s"' % self.backend._from_python(possible_value))

                query_frag = u"(%s)" % " OR ".join(in_options)
            elif filter_type == 'range':
                start = self.backend._from_python(prepared_value[0])
                end = self.backend._from_python(prepared_value[1])
                query_frag = u'["%s" TO "%s"]' % (start, end)
            elif filter_type == 'exact':
                if value.input_type_name == 'exact':
                    query_frag = prepared_value
                else:
                    prepared_value = Exact(prepared_value).prepare(self)
                    query_frag = filter_types[filter_type] % prepared_value
            else:
                if value.input_type_name != 'exact':
                    prepared_value = Exact(prepared_value).prepare(self)

                query_frag = filter_types[filter_type] % prepared_value

        if len(query_frag) and not isinstance(value, Raw):
            if not query_frag.startswith('(') and not query_frag.endswith(')'):
                query_frag = "(%s)" % query_frag

        return u"%s%s" % (index_fieldname, query_frag)


class ConfigurableElasticSearchEngine(ElasticsearchSearchEngine):
    query = CustomElasticsearchSearchQuery

In the above snippet 'contains': u'*%s*' is doing the job for us to allow single letter querying which provides better results. Add the ConfigurableElasticSearchEngine in your settings.py

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'myapp.backends.ConfigurableElasticSearchEngine',
        'URL': 'http://localhost:9200/',
        'INDEX_NAME': 'my_index',
    },
}

Now we are done with the customization. The customised searchengine's example search would look like:

SearchQuerySet().models(Book).filter_and(title=<query_string>)

Eg1: SearchQuerySet().models(Book).filter_and(title='m')
# Will give results of books with title 'mcrop', 'micropyramid', 'mycropyramid' and all other titles which are having letter 'm' in them.

Eg2: SearchQuerySet().models(Book).filter_and(title='mi')
# Will give results of books with title 'micropyramid'.

Note: Here __contains is optional as Haystack2.X added __contains as default for filtering the data.

Posted On 28 March 2014 By MicroPyramid

Previous post Next post

Need any Help in your Project?Let's Talk

Latest Comments

OUR SERVICES

OPEN SOURCE PACKAGES

Django-CRM :Customer relationship management based on Django

Django-blog-it : django blog with complete customization and ready to use with one click installer Edit

Django-webpacker : A django compressor tool

Django-MFA : Multi Factor Authentication

Docker-box : Web Interface to manage full blown docker containers and images

More...

Autocomplete with Django-Haystack and Elasticsearch with single letter querying.

Need any Help in your Project?Let's Talk

How to maintain user session across sub domains in Django

What is Django CRM and it’s advantages

Implementation of single sign on using auth0 in django application

Customizing Django REST API Serializers

How to implement TokenBasedAuthentication in DjangoRestFramework?

Understanding middleware functionality in django2.0

Django acceptance testing automation with robot framework

How to filter a Django Queryset using Extra

Overriding Django Model behaviour with Proxy Model

how to pass extra context data to serializers in django-rest-framework ?

Understanding Routers in Django-Rest-Framework

Django Single Sign On(SSO) to multiple applications

How to Document API Requests using Django Rest Swagger

Django webpacker - A compression tool to bundles css, js files

What's great about Django girls to inspire women into programming

How to use nested formsets in django

How to convert xml content into json using xmltodict

Custom validations for serializer fields Django Rest Framework

How To Add A Custom Managers In Django

Introduction to API development using Django REST framework with Example

Django-REST Framework Object Level Permissions and User Level Permissions

Using Python weasyprint generate HTML to PDF in django

How to create thumbnail image in Django templates using sorl-thumbnail?

Deploying Your Django app on Heroku

Django Template language Intro

Using Gitlab API, integrating gitlab in django project for authentication and access

Django Conditional Expressions in Queries

Sorl-thumbnail to generate thumbnails in django

How to Create initial django migrations for existing DB schema.

Django - Database access optimization

Querying with Django Q objects

Integration Of GitHub API with python django

Django - migrating from function based views to class based views

How To Write Custom migrations in Django

Dynamically Adding Google Maps with Marker In Django

Django Unit Test cases with Forms and Views

Understanding Checkout flow in Django Oscar.

How to monitor django application live events with sentry

Integration Of Linkedin API in python django

Django efficient implementation of Amazon s3 and Cloudfront CDN for faster loading.

How to create Custom User Model or Extend User Model in Django?

Working with Django Plugins

Export html web page to pdf using jspdf

Preserve file names with sorl for better SEO

Custom Password Less Authentication in Django

Improving page speed score in Google Page Score test- PART2

Extract text with OCR for all image types in python using pytesseract

Factory Boy - An alternative for Fixtures

Improving page speed score in Google Page Score test- PART1

Integration of 2checkout with django

Get count, average, min, max values from model field using Django Aggregate

Set Up Travis CI For Django project

Sending emails using sendgrid on heroku for a Django App

Django Custom Template Tags And Filters

Python Memcached Implementation for Django project

How to customize Django Oscar Models, Views and URL's

Integrate Django-Oscar-Accounts with Django-Oscar

Google plus API integration with django

Extract data from PDF and all Microsoft Office files in python

django Payu Payment gateway Integration

Running Django with PyPy to boost performance

How to Create your own e-commerce shop using Django-Oscar.

Setting Up Coveralls for Django Project

Sendgrid Inbound Email Parsing with django

How to customize the admin actions in list pages of Django admin?

Handling Custom Error Pages(404, 500) In Django

Autocomplete with Django-Haystack and Elasticsearch with single letter querying.

How to create Periodic Tasks with Django Celery?

Best Practices of Using Django Celery in Django Social Auth

Custom Decorators To Check User Roles And Permissions In Django

Implement search with Django-haystack and Elasticsearch Part-1

How to index binary files in django haystack

Understanding django serializers with examples

Introduction to Django's Class based views - Understanding how a class based view works

Understanding Django Permissions And Groups

Django Inclusion Tags

Dynamic Models in Django (adding new fields to models from admin)

E-commerce (paypal) integration with Django