Web Services

Designing HTTP APIs for slow operations

We got used to having everything on the Web, and so many backend operations became HTTP APIs. That’s also the case in the areas of AI and optimization, because it gives an easy-to-understand interface for clients that can be used without having to learn new technologies. There’s a problem, however: Some of these operations are very slow, and at times systems become a brittle mess of regular timeout hits, latency optimizations, and support issues.

Locating memory leaks in a Python service – Part 2: C-Extensions

Building up on last week’s issue about diagnosing memory leaks in Python, this blog post will show how to identify memory issues in native extension modules.

From [the first posting]({% post_url 2022-09-04-locating-memory-leaks-in-services-part1 %}) we have the tools to clearly diagnose a memory leak in pure Python. So let’s proceed to the next level and add some native code to the picture. It is one of the strengths of Python how easy it can be extended with compiled code e.g. in C, by so called “extension modules”. This is a key factor which made Python the first choice for data science: The heavy number crunching can be done by optimized routines and the program logic is written in the high-level Python language.

Locating memory leaks in a Python service – Part 1: Pure Python

Even in a garbage-collected language like Python it is possible to create applications with memory leaks. This means that the program keeps memory reserved vor variables which are no longer needed. So the memory could be freed, but it isn’t. Especially in long-running applications this can lead to a huge memory consumption over time, up to a point where the hardware limit is reached. In a series of two blog posts I’ll show how to debug such memory issues in a Python web service.

Envoy sidecar configuration generator

This post contains a configuration file generator for an envoy reverse proxy with all the bells and whistles. If you are looking for a working base configuration with the most important features of Envoy as sidecar proxy, this is the place to get one.

The Throughput Paradox – Why a Machine Learning API Needs Load Shedding

Especially compute-intensive services like a machine learning inference API can heavily suffer from queueing effects. I will demonstrate this with response time measurements of small demo service and show how to deal with this using load-shedding