Because Python has limited parallelism when using threads, using worker processes is a common way to take advantage of multiple CPU cores. The multiprocessing module is built-in to the standard library, so it’s frequently used for this purpose. But while multiple processes let you take advantage of multiple CPUs, moving data between processes can be very slow. And that can reduce some of the performance benefits of using worker processes. Let’s see: Why processes can have performance problems that threads […]

Do you want to build a compact visual progress indicator for your Python script using print(), but your output doesn’t show up when you’d expect it to? Or are you piping the logs of your script to another application, but you can’t manage to access them in real time? In both cases, data buffering is the culprit, and you can solve your troubles by flushing the output of print(). In this tutorial, you’ll learn how to: Flush the output data […]

The new year has arrived, and January brought a flurry of new and interesting Python enhancement proposals (PEPs). Topics range from f-string formalization and no-GIL Python to packaging. There’s been ample discussion on the Python Discourse forum about the implications of these PEPs, and if you’re still a bit wary of diving deeper into the discussions, then you can get a softer introduction here. There have also been a couple of noteworthy new releases, first and foremost the fourth alpha […]

Splitting a Python list into chunks is a common way of distributing the workload across multiple workers that can process them in parallel for faster results. Working with smaller pieces of data at a time may be the only way to fit a large dataset into computer memory. Sometimes, the very nature of the problem requires you to split the list into chunks. In this tutorial, you’ll explore the range of options for splitting a Python list—or another iterable—into chunks. […]

In this tutorial, you’ll build your own Wordle clone for the terminal. Since Josh Wardle launched Wordle in October 2021, millions of people have played it. While you can play the original game on the Web, you’ll create your version as a command-line application and then use the Rich library to make it look good. As you follow along in this step-by-step project, you’ll practice how to set up a simple prototype game before iteratively developing it into a solid […]

Libraries like NumPy and Pandas let you switch data types, which allows you to reduce memory usage. Switching from numpy.float64 (“double-precision” or 64-bit floats) to numpy.float32 (“single-precision” or 32-bit floats) cuts memory usage in half. But it does so at a cost: float32 can only store a much smaller range of numbers, with less precision. So if you want to save memory, how do you use float32 without distorting your results? Let’s find out! In particular, we will: Explore the […]

You have a file with data you want to process with Pandas, and you want to make sure you won’t run out of memory. How do you estimate memory usage given the file size? At times you may see estimates like these: “Have 5 to 10 times as much RAM as the size of your dataset”, or “several times the size of your dataset”, or 2×-3× the size of the dataset. All of these estimates can both under- and over-estimate […]

Most modern web applications are powered by a REST API under the hood. That way, developers can separate JavaScript front-end code from the back-end logic that a web framework like Flask provides. Following this step-by-step project, you’ll create an interactive single-page application with HTML, CSS, and JavaScript. The foundation is an existing Flask project with a REST API and a connected SQLite database, which you’ll grab in just a moment. In this tutorial, you’ll learn how to: Navigate a full-stack […]