How to solve the N+1 problem for non-DB requests in Django?
To remind, the N+1 problem means a code like this:
for comment in Comment.objects.filter(post_id=1).all():
print(comment.author.avatar_url)
makes N+1 requests to the database (hence the name):
SELECT author_id FROM appname_comment WHERE post_id = 1;
SELECT avatar_url FROM appname_author WHERE id = 1;
...
SEELCT avatar_url FROM appname_author WHERE id = n;
despite only 2 requests were really needed here (not considering a single-request JOIN or nested SELECT for reasons that will be clear later):
SELECT author_id FROM appname_comment WHERE post_id = 1;
SELECT avatar_url FROM appname_author WHERE id IN (1, ..., n);
Indeed, it’s a well-known problem and Django offers a simple solution to it:
for comment in Comment.objects.prefetch_related("author").filter(post_id=1).all():
print(comment.author.avatar_url)
But what if the “related” data isn’t stored in a DB table? For instance, if the user data such as avatars belongs to a separate service which is available over an HTTP API only.