Part of Python & Data — Python for data work, reporting and automation.
Most Python data tutorials solve one link in a chain. Read a CSV. Draw a chart. Write a PDF. Each one ends where the interesting part begins — when the file is not a clean CSV, when the join silently doubles your rows, when the chart has to leave your machine and land in someone’s inbox on a Monday morning.
This is the whole chain, in the order you actually hit it. Every step links to the detailed article, so you can drop in wherever you are stuck.
1. Getting the data in
Whatever we would prefer, the data arrives as a spreadsheet. Reading one is a single line:
import pandas as pd df = pd.read_excel("sales.xlsx", sheet_name="Q3", dtype={"order_id": str}) |
That dtype is not decoration. Without it pandas reads order_id as a number, drops the leading zeros, and you find out three steps later when nothing matches. Writing the file back so a colleague can still open it is the part that takes the afternoon — Excel and pandas, both directions covers sheets, column widths and keeping numbers as numbers.
Sometimes it arrives as a PDF instead, which is worse. Text extraction is solved; tables and charts are not. PDF data extraction with PyMuPDF gets the text and the table structure out. For the numbers that only exist as bar heights, a vision model is the honest answer — describing PDF charts with a VLM, including where it confidently invents a value.
Free ebook
Free AI Video, Generated Locally
Working scripts and measured benchmarks. Free.
No spam. Unsubscribe at any time.
And when the data lives in a database, the question is how much SQL you want to write. SQLAlchemy against PostgreSQL covers the ORM and, more usefully, where the abstraction stops helping. If you are still choosing where to put the data, creating a MySQL database takes the setup seriously — including the collation choice that costs you dearly the first time someone types an emoji.
2. Joining it without quietly losing rows
This is where most of the damage happens, and it happens silently.
before = len(orders) merged = orders.merge(customers, on="customer_id", how="left") assert len(merged) == before, f"join changed row count: {before} -> {len(merged)}" |
That assertion has saved me more time than any library. A many-to-many merge multiplies rows without a warning: the data looks fine, the totals do not, and you find it three steps later. Joining dataframes in pandas works through what merge, join and concat each do to the row count, with the join types drawn out.
3. Looking at it before you believe it
Plotting is not the last step. It is how you find out that the join was wrong.
A correlation heatmap is usually the fastest way to see whether the data makes sense at all — and the parameters are the part nobody remembers:
import seaborn as sns sns.heatmap(df.corr(numeric_only=True), annot=True, fmt=".2f", # numbers in the cells, two decimals vmin=-1, vmax=1, # fix the scale or the colours lie cmap="coolwarm", square=True) |
vmin and vmax matter more than they look. Without them seaborn scales colours to whatever range your data happens to have, so a weak correlation renders as bright red and you draw the wrong conclusion. Seaborn heatmaps, from first plot to publication-ready has every parameter in one table — it is the most-read thing on this blog, mostly because that table saves a trip to the docs.
For everything that is not a heatmap, the matplotlib gallery has hundreds of examples and you will use about ten. Ten matplotlib charts worth knowing is those ten, with a note on when each is the right choice — and when it is the chart people reach for out of habit.
4. Making it leave your machine
There is a moment in every data project where someone asks for it as a PDF. You can export the notebook and apologise for the margins, or you can spend an afternoon with ReportLab and never think about it again: building a PDF report with ReportLab — tables that break across pages properly, page numbers that count, headers that repeat.
Then it has to be sent, and this is where a two-hour job turns into an afternoon. The four lines work; the attachment breaks the encoding, the HTML renders as plain text in half the clients, and it all works locally and fails on the server because of TLS. Automating e-mail with Python covers all four.
I hit every one of those again recently while rebuilding the mail on this blog, so the article is not theoretical.
5. The tools underneath
Two things make the loop above faster, and both get skipped because they are not exciting.
The first is a debugger. Most data bugs are one wrong value in a dataframe, and stepping through beats scattering print statements — running and debugging Python in VS Code, including how to debug a script that needs command-line arguments.
The second is print itself, which does more than most people use. The print function properly covers sep, end, flush and f-string formatting — flush=True is the one worth knowing today, because without it a long-running script writing to a log file looks frozen.
Where to start, depending on where you are stuck
- The numbers are wrong and I do not know where — start with the join, then plot it
- It works but nobody can read the output — ReportLab, then sending it
- The chart looks wrong — the heatmap parameters, especially the scale ones
- I have not started — install Python, then VS Code
If a step behaves differently on your data than it does here, tell me — that is the most useful message I get, and it usually ends up in the article.
Free ebook
Free AI Video, Generated Locally
Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.
No spam. Unsubscribe at any time.
Leave a Reply