Can anyone point me to some really good tutorials that explain the logic behind when and how to split up a large Python program into separate files? Any best practices on where to do the splits? What are the pitfalls? What will I need to remember about how to use those files later in the main program that beginners always forget?
LambTracker Desktop is getting unwieldy and I need to make it more compartmentalized but not clear on how to best organize the code.
In Android Java every class would be a separate file but not sure if that makes sense in the Python world.
I donât have a reference but I have a practice in my coding.
In one case, I might split by function. Three example categories are UI controllers, general purpose utility routines, and core processing routines. By example, a function that sorts an input array is general purpose, a function that takes the dialog to define how to sort the array is UI, and a function that processes a sorted array for a specific output is core.
On a multi-functional development, I might also split by sub-function. For example, in an image processing tool that I am creating, I have a function base to load data files, a function base to calibrate the image scale, and a function base to remove the background. Each function base is in a separate file and gets included at compile time to the main routine.
Finally, one idea is to split routines in a way that you only touch the files with code that you need to update systematically while preserving (not touching) the files with code that is âstaticâ or âstableâ. Here again, you may have a routine to sort an array that you should never need to touch again compared to the routines that should process the results from the sorted array.
I donât recall seeing anything in the many books Iâve browsed/read on Python, but perhaps there is something somewhere.
My experience is that dealing with regular Python source file is easiest with a good editor that has Python-specific tools. I use Sublime Text most the time, supplemented by BBEdit. Sublime has some great ways of âfoldingâ the file at selectable levels of hierarchy. So, when âfolding allâ all you see is the first line of each âdefâ (or class). As I donât proactively split up Python source code files, Iâve seen one of the files in my most recent âprojectâ grow to ~12k lines, and I didnât notice until I just looked! the whole âsystemâ I have is about ~90k lines.
You could put each function/class in its own file, but feels to me too complicated and unnecessary (and adds complexity including stuff, I would guess).
That being said, if you are using Python with a particular framework, e.g. Django (which I like and use, not for web sites but for itâs terrific way of handling queries to a MySQL database), then to best take advantage of the framework they usually do provide guidance. Django documentation provides guidance and Iâm sure there are many other web sites and books, but I wonât pursue here as I donât know if you are using Django.
All that being said, if you must break it up, Iâd say do it by âfunctionâ. And I would probably move stuff that is âdoneâ and tested into so-called âlibrary.pyâ that holds common stuff.
I know you are looking for more specific guidance, but Iâve never run across it.
The principles in this book will help you. It is not language-specific, but conceptual. It changed my programming life, and has allowed me to write and maintain an open-source application for the past 17 years with very few bugs, eliminating things âgoing southâ when a change is made, minimizing/eliminating âchain reactionâ type events where a change here mandates a change there, etc.
I follow the same principles I learned as a C++ developer, every class is a separate source file and every class should be small and have a well defined purpose. Donât be afraid to create a class with just a few methods. Your computer can handle a project with hundreds of source files no problem.
My app is designed about a series of screens or windows. each one deals with a particular subset of functions. For example right now I have Animal Search, Add/Edit Animal and Animal Reports all as very separate things on separate Toplevel tkinter windows. Population Analysis will result in several more screens and so on for major categories of stuff. Within a section, like Animal Search most of the code is only used there and to navigate to another section you go to a completely new screen.
Iâm using PyCharm as the IDE so not an editor per se. I havenât looked at folding possibilities at all, thanks for mentioning that.
Nope, standard Python 3.7 with tkinter and ttk as the UI toolkit. Trying for minimal excess stuff although I know I will be adding SciPy and Numpy once I get into the population analysis code.
Thanks for the book suggestion. I just got the sample on kindle and it looks like a lot of the conceptual stuff I am looking for. As an aside, whatâs your open source project? LambTracker is also open source, none of the desktop code is out available on my GitLab repository yet but should be RSN I hope.
I appreciate that observation as well. I will be packaging the final LambTracker into a package for Mac, Linux and Windows probably using PyInstaller although exactly how isnât yet decided. Goal is for the inexperienced user to be able to just run the app and all the python interpreter, modules and libraries etc are included. Most of my potential users wouldnât know how to install a Python interpreter or load something like SciPy at all so I have to have it appear as a monolithic application to the end user.
If planning to use above, then be sure to look into and seriously consider PandasâŚ
âTrying for minimal excess stuffâ
⌠I find relying on Django (for doing database interface) simplifies and (massively) reduces size of the app that I have to program with a lot of additional capability. And de-risks as less to test/debug.
The usual way of making printed circuit boards is to chemically coat the copper, expose to light through a mask (often created with a laser printer on transparency film, sometimes ironed on, sometimes hand drawn), remove excess coating, then etch with ferric chloride (for instance) to remove excess copper, use another chemical to remove the remaining coating, then drill holes for mounting through-hole components, etc.
I got tired of ruining t-shirts with the chemicals, and making bad boards, so I wrote a plugin for a printed circuit design program called EAGLE that creates g-code (x,y,z coordinates and tool control code) that drives a computer-controlled router to cut out around the tracks of the printed circuit board and drill holes for components. This eliminates the need for toxic and messy chemicals, and automates the processing of drilling holes, which people usually do by hand. My program is in use by, I estimate, a few thousand people around the world.
Typical router, though this varies widely to include industrial CNC milling machines
Iâm ignorant of what Pandas actually does. Iâm going to be calculating coefficients of inbreeding, kinship/relationship, genetic distance, generation interval, breeding population etc.
Isnât Django only for building web apps?
LambTracker is stand alone. Plan is to have it run as an ordinary app on Mac, Linux and Windows. My database is SQLite. For me the database part is easy, itâs the darned GUI that is a PITA to develop! First off delete never happens due to requirements for maintaining a history. I add records and I occasionally update records to fill previously empty fields but records do not get deleted.
Pandas is brilliant. Itâs like a merged sql, numpy, and ORM. Makes traversing and manipulating large datasets require very little code and mostly performant.
For code organization, I havenât been in charge of a codebase large enough where organization really mattered. But, at work the part of our codebase in python is separated be features. (In terms of files and directories).
Like with anything else With regards to organization, I think the most important thing is just to be consistent.
Django used for web apps, but thatâs not all itâs good for. I express my database model (can use all the leading dbâs including SQLite) in Django which then makes it Python-friendly âobjectsâ. Comes into its own for relational databases having numerous relationships (most data models do).
Neither may be for you, but if i were doing what I see you are doing, Iâd do it on top of Pandas and Django.
