I have ADD and I'm struggling to write a more complex Python app right now. I'm hating myself for not being able to easily skip around the code as it gets more complex with more classes and methods. So this is very apropos for me.
Thing is, I can skip around Terraform much more easily (in my normal DevOps role). I'm wondering if Python whitespace isn't "right" for me somehow. EDIT: I'm definitely more used to Terraform, but I can read and navigate nodejs far more easily than Python, it's weird. I also don't want to come across as blaming Python for this, certainly it's incredibly effective for a huge number of people!
My next step is to try to break out the classes into their own files or something like that.
I don't suppose anyone has any tips on managing Python scripts? I doubt this is even 2,000 lines yet :(
As someone on more or less the maximum dose of adderall, here are the commandments as I have been able to divine (for myself):
- Thou shalt use a language server. Esp. goto-definition and find-refs, as they shall light your path through third party libs.
- Thou shalt refactor like a crazy person. No component should have less than 3 or more than 5 significant members. A concise codebase makes for a calm mind.
- Thou shalt use black formatting, and obey *all* recommendations from the linter. Thus shall the structure of your code always explain it's function.
- Thou shalt use mypy, as it is The Way.
- Thou shalt follow the way of the Unix, creating isolated components that Do One Thing Well and define concrete APIs between them. Thus shall you both tame the Spaghetti Monster and save bits of your work from the maw of the great destroyer This-Is-Too-Complex-So-Let's-Rewrite-The-Whole-Thing.
2: This I need to do. I wanted to use the Traitlets module for my config parameters/data so I need to refactor that part of the app.
I did wonder about something like a TerraformCloud class because so far I have a bunch of methods in there: search_workspaces, get_workspace_id, get_variable_id, is_workspace_used, get_unused_workspace, enable_workspace_module, set_workspace_as_used, set_variable, start_workspace_run, and probably more in future. Should I be breaking those up into TerraformCloud, TerraformWorkspace, and WorkspaceVariable? Sorry for the question, I am still learning, and the terrasnek module is helping a lot with all this Terraform stuff.
3: I do use the black formatter. The only thing I question is its line-length rule.
4: I didn't know about mypy, that's pretty great and I'm super grateful!
5: Yeah, I am probably failing hard at that one. I will try to return to The Holy Path Of Unix.
Thank you for all of these. I'm so very grateful for your help!
Given the mention of great destroyer This-Is-Too-Complex-So-Let's-Rewrite-The-Whole-Thing, you seem like you know what you're talking about, but I can't comprehend the actual lesson you're trying to impart.
I am a professional Python developer. Having a good LSP is critical but it doesn't fix Python's biggest problem - the space scoping. It never gets easier you just eventually "get used to it". Keeping methods to a size that can fit on screen helps a lot but on larger projects this can be a lot to ask for. An LSP + a way to mark code is usually how I like to do things.
Your criticisms are valid. Python is the flavor-du-jour and often shoehorned into places it doesn't work well. One of those places is scaling with lines of code. It just simply begins falling apart as a nice experience once projects reach 10,000-20,000 lines. The space scoping becomes much harder to deal with and cognitive load increases in a hockey stick way after a critical line count. Just wait until you try to remedy some other problems with MyPy only to realize is just a stronger linter and not an actual type system :).
2000 lines in a single script file is a lot to mentally process. Unlike a large utils.py (the junk drawer of a putting module), with a script there is often a wide range of functions and classes.
My best advice would be turn this into a more organized package that you install with pipx[0] on the systems that need it. The click[1] package is helpful too.
Learn to use search tooling. My neovim setup has ripgrep + a LSP on hand at all times. You can do the same things in VSCode.
If you don't know where something is... search, search, search. 2000 lines or 2 million lines. Searching cuts the mental load because you find what you NEED at that moment.
Need to know what refers to this function or variable.. there's a click or key sequence for that.
And yeah, Python is a bit of a trip to read, the lack of braces makes it unique to parse, if you aren't used to it.
And yeah, Python is a bit of a trip to read,
the lack of braces makes it unique to parse,
if you aren't used to it.
Huge thank you for this. Just hearing that validation is helpful :)
Good point about searching. I've been using vscode, which I use for Terraform a lot and jumping around with the outline view. I need to figure out if there's a way to search in vscode and edit code while keeping my hands on the keyboard.
Or use neovim, because my beloved Emacs is often slow.
Thing is, I can skip around Terraform much more easily (in my normal DevOps role). I'm wondering if Python whitespace isn't "right" for me somehow. EDIT: I'm definitely more used to Terraform, but I can read and navigate nodejs far more easily than Python, it's weird. I also don't want to come across as blaming Python for this, certainly it's incredibly effective for a huge number of people!
My next step is to try to break out the classes into their own files or something like that.
I don't suppose anyone has any tips on managing Python scripts? I doubt this is even 2,000 lines yet :(