Although it's been posted here (only twice, though!) before, Philip Guo put up a 10-hour series of lectures on CPython internals[0]. I am nearly done with them, and have found them very useful and well-paced. Posting here because the author of this article seems to have stopped the series, and anyone interested in this stuff should definitely check out the Guo lectures.
Thanks everyone! Recording lectures has been great since it doesn't take much extra work than not recording; just takes some time to render the final videos and upload. Ideally I'd have time to split the videos up into smaller chunks, but alas free time is diminishing nowadays.
Would you have any interest in providing someone (I would volunteer) the videos so that they can package them in smaller, topic-specific pieces for easier consumption? I don't know anything about editing videos, but if your writings and lectures have taught me anything, it's that I'm capable of learning things and using that knowledge to do interesting and useful work.
As an aside, you are one of my main sources of inspiration. I was a high school dropout with several terms of failing out of community college courses, but after reading your PhD Grind series/book, I'm now a (nearly) straight A student and am very determined to go as far with my education as I can. I would be honored to help you in any small way I can to express my gratitude!
Thank you for your effort and sharing the knowledge. pythontutor.com is excellent project!
While watching the series for second time - I started taking some notes/hints. Here its: http://giis.co.in/c_py_notes.txt
Its more like hints. one need to watch video to understand this small note. (sorry for typos/mistakes)
thanks again!
EDIT: with youtube offline mode, I saved some video on smartphone & then while going for evening walk, I was listening to it, Got better in health & python skills at the same time :D
Contrary to what the article says, PyVarObject is still PyObject at the same time and PyVarObjects have nothing to do with mutability (certainly, it's trivialy obvious that not all PyVarObjects are mutable as immutable tuples are PyVarObjects, while mutable lists are not)
The thing that is handled by PyVarObject are types whose instances can differ in their size, with tuple and bytes objects being probably most common cases when it is used.
If you try to embed or extend Python in C you will be exposed to PyObject very quickly. The official Python documentation can walk you through that process (visit the embedding or extending parts).
The PyObject structure mentioned in the article also comes with many functions to manage them programatically, cast them to other types, etc.
I'm curious if you know much about jobs doing this kind of thing. I've worked a ton with Python, and know a lot about using Cython to write extension modules, wrap C++ code, etc.
I really want a job where I find performance bottlenecks in research or business analytics or financial analytics code, migrate parts to robust Cython implementations, and then build and maintain the broader Python libraries and APIs surrounding that.
But what I'm unfortunately finding is that basically nobody uses this stuff anywhere in practice. I've interviewed at a ton of places doing everything in C++ entirely, and I just see their code and I think, my god, why would anyone choose this? You can write the critical stuff in C++ just as you are, and then so easily wrap it for Python, then write the other 99% of the code in super easy, dynamic Python, even with various kinds of validating systems, or using typing and type annotations to avoid certain kinds of bugs, and just everyone's life is easier.
But they are just so entrenched in the old way of doing it that nobody is willing to try.
I've been consistently amazed at how few jobs in the monthly Who Is Hiring thread are tagged even just with NumPy. I think I've seen maybe one job in 6 months that mentions Cython, and then I found two others outside of HN and the interviews with those places didn't work out.
How to find a job doing this stuff is like my main vexing work problem at the moment. It's so useful, but it's sort of locked outside of some kind of energy barrier that quant teams seem unwilling to cross.
Suggest you look for an opening on JP Morgan's Athena project, or Bank of America Merrill Lynch Quartz. Both those platforms are C++ on the inside with Python APIs. The originators have gone independent here: http://www.wsq.io/about-us/ The site says they're hiring.
I don't know why that throwaway account's comment below was removed ... it's ubiquitously well-known that the code quality inside of the Quartz/Athena/other spinoffs at other banks silo is utterly terrible, like don't go within ten feet of it without a radiation hazard suit on.
This is not a controversial opinion, nor really even a swipe at anyone. It's just bad, bad, bad code inside of banks.
I remember talking to someone on the Athena team during a job interview there and they told me that they had to fill out paperwork and wait weeks for approval just to pip install a third-party package for a sandboxed and not-production-facing ad hoc data analysis task. I don't care what industry you're in, that's just taking bureaucracy way too far. I've worked in quant finance before in a highly regulated environment, and there was never the slightest worry that you couldn't just grab any third-party tool you needed to do your job. Getting into production would require a big process, sure, but not just grabbing pandas, say, and answering someone's quick ad hoc business question. In fact, this sort of thing also was never a problem in defense research either.
Anyway, I personally view the fact that the team who spun out of I believe Goldman originally to make the Athena/Quartz/etc stuff (I think now they are even infecting smaller banks like PNC with it too?), is easily one of the worst things that has happened to the world financially in the past 15 years. Sadly not hyperbole.
I don't think I would be a good match for bank culture or HFT, but if the work you describe is in a small-to-medium sized asset manager or hedge fund, then I would not mind, and would find it enjoyable. I worked at such a place a number of years ago and it was one of the better jobs I've had.
The trouble I find is that a lot of small-to-medium sized asset managers and hedge funds don't have very good technology practices in general, and are pretty far from sophisticated use of low-level Python. There are often a lot of political reasons why they cling to Excel/VBA, R, or MATLAB, and there's almost always some political battle happening between the people that want to rationalize the system to a proper software design, and people who just want to keep cranking on the hodge podge of existing tools.
I'd probably be fine with some trade-off regarding all of that, if the pay was acceptable and there was a strong commitment to a healthy work/life balance. But among finance firms this is extremely hard to find.
So even though I am not at all opposed to doing this work in finance, I still find the volume of acceptable job openings in that field to be too small to make it realistic.
I have worked on two of these. I have seen the code for all three. These systems have too many WTFs to count.
They have lead to technical crisis at both JP Morgan and Bank of America Merrill Lynch. At Bank of America Merrill Lynch, the crisis lead to resignations of the chief architect and the CTO.
Well, people have found success embedding Lua or JavaScript, since they're easier to sandbox than Python.
Now, Python extensions can be a bit hard to get right. Especially since Python has a specific threading model and if you are trying to wrap an existing C++ code base that is multithreaded in Python... it can be challenging.
Python offers a standard library with many, many functionalities. You can start processes, do networking, manipulate files, etc. You might not necessarily want to give your scripting environment those facilities.
I worked remotely for Continuum before. As I mentioned in another comment, I think Continuum is a great place with amazing engineers, but my particular working style was not well-suited for the mixture of start-up environment and ad hoc consulting. My projects also did not happen to feature the use of any C stuff, only database stuff, pure Python, and pandas. I do think some other projects involved the Cython stuff, but nothing I was assigned to work on.
So unfortunately these types of options aren't likely a good match for the sort of work I am looking for.
I do all of the stuff you mentioned on a daily basis (quant in a high-frequency trading firm) -- NumPy (plus its C API), Cython, numba, a good measure of C++, and even some occasional LLVM. The typical projects range from ingesting market data in high volumes to doing complex machine learning mumbo jumbo :)
I've tried applying at SIG in Philly in the past; I have all the things you mention on my resume (I worked previously at Continuum and also at a quant asset manager), but have not ever heard any kind of reply to my resume (not even a rejection) unfortunately.
It's funny because I do work at SIG (Dublin, Ireland), and we do know a few folks from Continuum personally.
For a quant role, even if in the end it is likely to involve a lot of hardcore low level coding, you're (generally) expected to have a PhD in math/phys/stat/eeng though, that's the rules. For developers roles I don't know all the details. But one thing for sure, we're hiring for quant roles pretty aggressively right now :)
When I worked at a large quant asset manager as a quant analyst in Boston, I did not have a PhD and neither did any of the other analysts except maybe 1 or 2. It did seem you needed a PhD for portfolio management or higher, but not for quant analyst. This was also true at a hedge fund in Boston ... I ultimately turned down their offer, but they seemd to mostly hire master's degree folks (I have a master's in stats/machine learning) not full PhDs.
Are there any openings (that would sponsor a visa) in the Ireland office? I'd love to apply and think SIG could be a good fit, but I've just had no luck responding to the Stack Overflow ads from the Philly office.
I actually worked at Continuum already. It's a great place with some of the best Python engineers you can possibly find.
I found that the particular combination of start-up environment and consulting was not a good fit for me, and I suspect I would feel the same about Enthought or other kinds of analytics consulting companies, so alas this is not an option for me.
Cython can be used to auto-generate optimized Python C-API code that gets source-mapped line by line to original Cython code in the form of HTML annotations. This stuff is very hard to write by hand.
[0]: http://pgbovine.net/cpython-internals.htm