I find this super interesting!! while having a quick look, I got the impression, that Mercury runs 1 web service for 1 notebook right? For me it would be very helpful to something like a dashboard with 1 webservice supporting n notebooks.
You can publish multiple notebooks on 1 web service. There is no limit. We provide default view where all notebooks are displayed as cards with clickable title. Please check example in the documentation https://runmercury.com/docs/multiple-notebooks/.
I'm using Papermill to operationalize Notebooks (https://github.com/nteract/papermill), it e.g. also has airflow support. I'm really happy with papermill for automatic notebook execution, in my field it's nice that we can go very quickly from analysis to operations -- while having super transparent "logging" in the executed notebooks.
thanks for that! we're all on-premise with our infrastructure, so I probably won't be allowed to use AWS here (I could imagine that the aws parameter store is also even more helpful in an architecture with several aws components)
yo, i work in the energy industry. it's not quite as terrifying as it looks. energy retailers typically sell electricity to end users at a fixed price. prudent retailers _hedge_ this exposure, i.e. they buy the electricity on wholesale markets that they promised to delivery to you at a fixed price, in order to reduce their risk to wholesale prices going up.
wholesale price went up (A LOT) which drove many retailers across europe into bankruptcy, but well - they didnt hedge so well..
if you want a proxy for retail electricty prices is worth checking the wholesale markets, you can do that yourself on EEX's website: https://www.eex.com/en/market-data/power/futures
you'll see the following prices for trading day 2021-12-29: Year 2022 settled at 225EUR/MWh or 22.5c/Kwh -- which is admittedly crazy high. But year 2023 settled at 128 EUR and the following years below 100 EUR. This is plausible as we the high prices we have now provide a strong incentive to build new electricity generating facilities in the future. and wind and pv are already profitable at prices 50-80 EUR/MWh without extra subsidies.
lastly: there are still enough retailers selling you power at prices below what your retailer wants to charge you, check https://verivox.de/ or the likes for alternatives
i've used prophet along many other ts methods for price forecasting in energy trading. my experience is that prophet is ok, but rather opaque. having tried many packages, I've always come back to the classical statistical methods seeing benefits in transparency (what's actually going on? impact of regressors?), speed and most importantly that these methods force the user to think about what's happening in the data and make conscious decisions about how to model things.
but i can see that sometimes you dont care too much about accuracy and understanding but that you just want a forecast for something that works decently without much hassle.
I work in energy trading in Europe and for us this is a blessing. Of course we store our time series TZ aware, but many less professional small utilities or other market participants (incl. Grid operators) who e.g. Order power on hourly resolution with tz-unaware excel spreadsheets via email are a nightmare as each participants treatment of double or missing hours is different and requires manual processing.
> Why don't you just make a system that the clients must enter data through? Instead of accepting spreadsheets...
It’s cheap to train people to fill out a spreadsheet. If you force people to use a custom data portal, you’ll likely increase error rates and give your counterparties a reason to prefer other shops.
Working in the same sector, I can confirm this. We try to offer pretty much any half-way reasonable interface to our clients for that exact reason, be it spreadsheets or csv via mail or SFTP, REST APIs or a web interface.
yeah, it's both what you say and sales people's poor understanding of how annoying that is (and the fact that they get rewarded for closed contracts where operational costs dont matter)
I did some work in this sector about 10 years ago. We wrote a system to parse these spreadsheets, and convert the various units of measure. At the time, we were processing the data faster than anyone else and used that data to our advantage.
The reason we couldn't mandate our own system was that many facilities are jointly owned, and we only had the primary ownership of a minority of facilities.
I can imagine. In my country the electricity meters only deal with winter time (UTC+2). Most suppliers have different rates for day and night, so in winter night rates start at 12am and in summer they start at 11pm (or is it the other way around?) so nothing needs to be changed on the meter.
To my knowledge, you won't loose power because your energy provider forgot to buy energy from the market, they still have to pay for it though (probably with late fees).
It depends on the market design of course, but generally, if too many energy providers did not match their consumption with generation (or bought power), that could lead to power outages.
Concerning the 'late fees': if your consumption exceeds your production, you pay what is called balancing energy. Essentially, you have to buy the energy for a price that is set by the transmission system operator depending on the total imbalance in the grid. In some cases, this might actually be beneficial to the energy provider as the balancing energy is sometimes cheaper than the prices you get on the sport market close to gate closure.
I like to connect with other devs working in the same industry as me. I work for Contigo Software, we make ETRM software and various other things for the energy industry.
There's various ways to contact me in my profile if your interested in chatting.
The other day I tried to delete my udacity account. No button in settings, I mailed support, they asked my to mail legal (apparently they don't fwd mail..). From legal I received the following response:
Hello,
We are in receipt of your request to delete your account. We are sorry you want to leave the Udacity family; you will be missed.
Before we can proceed with fully processing your request, we need to verify your information. We take these steps to minimize risk to the security of your information and of fraudulent information and removal requests. Specifically, we ask that you provide the following pieces of information:
- Username and email address associated with your Udacity User Account (if different from the one you provided in your initial request);
- Online Courses currently or previously enrolled in;
- Approximate date of User Account registration;
- Country of residence; and
- A statement under penalty of perjury that all information in your request is truthful and that this is your User Account or that you have the authorization to make the request on behalf of the owner of the User Account.
If, after you have provided the above information, we are unable to verify your identity and/or authority to issue the request, we may reach out to you for further verification information.
As a reminder, any deletion actions we take in response to your request are not reversible and may result in Udacity (or you) being unable to retrieve information about your account, enrollment, and records of completion. Please also keep in mind that all removals of such information are subject to requirements to maintain certain data in our archives for legal or legitimate business purposes.
If you have any questions about this request please see our Privacy Policy or let us know.
> A statement under penalty of perjury that all information in your request is truthful and that this is your User Account or that you have the authorization to make the request on behalf of the owner of the User Account
That strikes me as really extreme. If I had an Udacity account and someone tried to delete it, I would be annoyed, but I wouldn't seek to press criminal charges against them.
Their concern is probably someone deleting somebody else's account, along with all record of courses completed, etc. Is it possible to have an Udacity account if you're not actually paying them? If so, "account removed" and "unpaid account" are very different things and the transition between them is one-way.
> Their concern is probably someone deleting somebody else's account
Yes, that's what I meant. If some stranger deleted my account, I don't think I would be angry enough to attempt to press criminal perjury charges against them.
It may depend on the real-world relevance of the courses you've taken.
What happens if you're taking part in the Georgia Tech/Udacity online Master's in Computer Science, and suddenly all of your course history on Udacity is gone? Are you then stuck with remotely trying to access records through whatever GA Tech has retained for MOOC students in a degree program? Does it prevent you from signing up for additional courses because there's no record of you completing the prerequisites? Does their platform have a provision for overriding those prerequisites, and what would that have to be based on - word from a professor who taught a MOOC, has never seen you or your work and probably depends on Udacity's listings?
A similar situation could arise if the platform is being used for mandatory continuing education credits in fields that have that. An industry that has CE requirements would often be well-served to partner with an established and stable online learning platform rather than building their own. If you're required to take X hours of CE annually to retain a license but the only proof you have is a PDF certificate from someplace that says "We've never heard of this person," do you qualify or do you end up in licensing/certification hell?
It depends on what an account deletion means, yeah. Is there a GDPR for universities that makes them delete all history of your attendance at that institute? Do customer privacy laws force Udacity to do a more thorough deletion than you might expect if you're treating it like a website rather than a university or vice-versa?
IANL but I think it's perjury to lie in a declaration "under penalty of perjury" only if that declaration is specifically required under the law or if you are submitting it to a legal body, such as a court, or to achieve an expressly defined legal outcome. For example, the DMCA specifically requires such a declaration when making a complaint, so private parties can require it and you could face criminal prosecution for lying.
On the other hand, I have no idea what legal justification could Udacity claim. It's probably bogus.
If I had a nanodegree and was listing it on my resume or mentioning it in job interviews, I'd be pretty upset if someone came in and deleted all proof that I had completed it.
Yeah, I'd definitely be pretty upset if someone wiped the record of my college degree, but (I hope) Udacity courses don't cost nearly as much as I had to pay.
Depend probabbly not, but "hey there is this thing this guy did" might be a real thing. Probably not weighted as much as other things but still relevant to some.
But if it's a simple "hey there's this thing this guy did", are they going to go to the trouble to actually verify with Udacity that the person took the course? Probably not.
This is unacceptable - if an authenticated login is good enough to change account details and sign-in info, why isn't it good enough to close an account?
I'm not an Udacity customer, and after reading your account I never will be - I loathe these scummy tactics.
Many people (me included) hold the opinion that you should be able to control your data, which includes the right to ask a company storing data about you to delete it unless they have a valid justification not to (e.g. they may have to keep payment history for 10 years for compliance reasons).
In fact, in Europe it's so many people that that's the law.
Wow, thank you so much for bringing this to light. Our org is literally in the process of signing up for their enterprise product but I can not in good conscience allow it to proceed after seeing that, it is unprofessional on so many levels. I would expect this kind of contractual bullshit from an NY landlord, not from an education platform.
I know they wont miss our 220ish users, but I hope enough individuals see your post to put a dent in their numbers.
Protecting completed educational certificates for their users free of cost seems reasonable. It's not like they force you to keep paying, they are just not easily deleting your educational history on their platform.
It's worth mentioning that people are completing courses, nanodegrees, and even Master's degrees through Georgia Tech on Udacity. These are significant life events for people trying to change/shift their careers, get that next promotion, etc. Having all records of your Master's degree deleted by a 'griefer' is a terrible thing to have to recover from.
I have no affiliation with Udacity but I feel that yours is an extreme response to a company that holds records of work that is very dear to some people and their careers.
> As a reminder, any deletion actions we take in response to your request are not reversible and may result in Udacity (or you) being unable to retrieve information about your account, enrollment, and records of completion.
Didn't windows 95 popularize the idea of the trashcan - temporary deletion - 20+ years ago? Don't many db tools support the idea of "soft deletes"?
"Hey - once we do this, you can't get anything back, ever, immediately, it's all gone. For good. But we're going to keep stuff forever that you can never see, and we won't tell you what it is. But you can't have your stuff back, ever. It's gone."
Even my "offline" university had the idea of "not paid", "suspended", "active", etc. Keep me on file, and I'll pay again in 12 months when I take more classes, etc.
These days if I want to "delete" an account, I'll just change my email address and password associated with that account. I take help of temporary email providers. Also, I write gibberish in place of my profile data.
I wasn't actually paying. The account basically dormant and from back when they started and courses were free. I am just trying to minimize my digital footprint by deleting accounts I don't actually use (I'm happy with value for money at coursera).
That delete process sounds like a pain, but I suspect even a "reasonable" delete process is gonna involve a lot of steps and be somewhat a pain considering the consequences of actually deleting data.
fyi - given it's annoying to collect all that information, I felt udacity might actually be able to help me with that. So I filed a subject access request under GDPR and asked them to provide me with all the data they have in relation to me..
