The project is very interesting, but I'm not sure why they are doing it. How is that protecting user rights? This doesn't measure AI/ML progress that's available to state actors.
> EFF’s interest in AI progress is primarily from a policy perspective. We want to know what types of AI we need to start engaging with on legal, political, and technical safety fronts. Beyond that, we’re also just excited to see how many things computers are learning to do over time.
> Given that machine learning tools and AI techniques are increasingly part of our everyday lives, it is critical that journalists, policy makers, and technology users understand the state of the field. When improperly designed or deployed, machine learning methods can violate privacy, threaten safety, and perpetuate inequality and injustice. Stakeholders must be able to anticipate such risks and policy questions before they arise, rather than playing catch-up with the technology. To this end, it’s part of the responsibility of researchers, engineers, and developers in the field to help make information about their life-changing research widely available and understandable. We hope you’ll join us.
Yes, it's not that I didn't read the page. But even in case of corporations, it's not really possible to gauge what t's available to them, because they don't share everything.
Well on the other hand whatever is available publicly, they have at least that. So maybe the whole research does make some sense in this context.
Perhaps forcing governments and corporations to share more will be explored if it seems that this publicly documented baseline is far from the state of the art. Impossible to gauge, perhaps, but even realizing that highlights the risk of public ignorance.
A question mark is more valuable than simply reporting nothing.
And I think as the Samsung / Intel / semiconductor fab article (about birth defects and cancer caused by chemical exposure) the other day noted, traditional lawmaking is ill-equiped to deal with fast-moving technical fields.
Algorithmic sorting is a way to circumvent privacy protections by laws enforcement. For instance: "I only capture the emails that contain the word Al-Quaeda inside". For this an algorithm is fed all the emails of a given ISP but only selects a few.
Aren't you interested in knowing the capabilities and bias of the algorithms used to filter out these data?
Also, there is another important question in AI: what is the status of the neural networks generated through feeding some other data inside? Is it a derivative work? If you feed an AI some GPL work does it make the resulting NN GPL too?
Looking from within the AI/ML field, I don't think that state actors (or the major tech megacorps) are somehow special in terms of progress in a manner that makes them unmeasurable.
Yes, they have certain datasets that are somewhat larger/better than those available to the research community and somewhat easier access to larger clusters of hardware. However, that's generally a minor and (more importantly) predictable advantage - I may not be able to train as good a model of X as Google does because they have more data and hardware, but I can estimate that their model is likely to have e.g. 20% lower error rate; which gives a meaningful advantage in product quality but doesn't really enable anything conceptually different.
Yes, there are certain technologies that are developed by them and not yet published, but looking at all the competition, what is seen in the products they build, and and what is seen in what state actors are funding/attempting to buy, they're not significantly beyond what is known to the public, at there seems to be no more than 6-12 months lag between when a capability becomes available and when it's seen by everyone else. It may take more than that for others to replicate, but within 6-12 months, people would have a general idea on what it'd take to replicate and what are the limits of such capabilities.
Yes, the bigger actors may have invested lots of effort to polish things that are important to them - so if you're seeing a researcher's proof of concept held together by duct tape then you should keep in mind that somebody can have a reasonably working prototype of that hidden in their labs; but if you're not seeing anything close to a proof of concept in the research community (or at least claims "we know how to make a proof of concept quickly if there was funding for that"), then the capability doesn't exit yet; somebody may have unpublished results but that's included in the "6-12 months advantage" I mentioned above.
Regarding user rights, IMHO you don't need to worry about still unknown capabilities of ML/AI but instead about how (and how widely) they'll apply and deploy ML/AI capabilities that are already well known. E.g. there's no question that we can run face recognition to everyone seen by a sufficiently high quality camera and try to match those places to social media profiles; all serious state actors definitely have such a capability iff they are willing (and have the funding/manpower) to build and run the required infrastructure. Whether they will do so is a question of economics and politics, not about AI/ML tech. In order to massively (ab)use something, it needs to have a certain technological readiness level; and there's a long gap between invention and sufficient maturity for that.
Part of what we are learning, in AI/ML, is the number of problems where the ML is relevant and where it is not. (Or, to put it another way, where we can find statistical relevance between features and targets).
So I think you need some kind of "meta-metric" that measures the growth of the taxonomy itself. And perhaps some kind of weighting for the impact of the solution.
There is also an interaction effect (for instance, Natural Language Processing is powerful, and "common sense reasoning" is powerful, but put them together and you have a knockout), but I don't know how to go about measuring that.
I think you'd have to measure "common sense reasoning" first. With NLP you can have accuracy/precision scores. With common sense it's almost equivalent to "what you need to beat the turning test" and not easily measured. I think if you get that part you essentially have the NLP. In that those last few percent error of NLP are generally "common sense" issues.
Who is, or is seeking to be, the canonical source of truth for AI, ML etc policy/position/etc and why should/how do we trust them?
In twenty years, what body will be directing the policies and laws regulations etc WRT to how humanity deals with essentially what is another "sentient" species?
Edit.. just read this and apparently this is exactly what the EFF is attempting to do...
But the question still remains: how do we trust these policies, how do we request/reject them?
I don't want to deal with this the same way the legal system is currently set up, lawyers and the law is flawed in many respects and I don't think it's a good idea to map the old to the new and uncharted directly.
(Apologies for the clunky language/terms.. please educate me on how to speak of this if you know)
EFF isn't necessarily a fan of the way that current institutions of governance or the law operate, but when those institutions attempt to interfere with the development of technology, we step in to try to mitigate the damage and make the case for sensible outcomes.
In the case of general-purpose human level AI, which to be clear is an extremely speculative kind of technology that might not happen in our lifetimes, I don't think anybody knows how humanity would deal with it. If it does happen, I think the biggest responsibility of participants in that process would be to minimize the risk of instability and conflict while humans and the new species (possibly species, plural; possibly not a species at all), figured out how to relate to each other.
How best to accomplish that is largely a very difficult and mostly unanswered research question, though you can find some pointers to some interesting early work in the safety section of the Notebook.
If we cant determine when/if we will get on parity with humans in AI capability... we should be focused on AI as a tool that will be used by those with the know-how-resources against those who do not have ability/access;
Rich vs poor:
* HFT will never be a common man's tool
* Government surveillance
* Corporate surveillance
* Behaviorally informed/adjusted pricing etc...
Basically ML and AI will be used, in large part, by those who can to exploit those who cannot (or at least those who wont) defend themselves.
There will be no opting out.
So, with that said, if we go from the bottom up - and there is no opting out, then what is the best that one can hope for? I'd say complete ownership and control of one's own "meta-cloud" -- Any data that is a resultant trail of any action I take as an individual should be owned by me, and I should be able to see it all, and delete or block it.
Or, in the extreme case, shouldn't I be able to require that any information presented to me (such as an ad or a price) be required to inform me as to how that information was formulated:
"You are seeing this price because the following factors were analysed..."
The real question is, in the future, is there even such a concept as "off grid"
For ease of reading/UI, I think they should use a different color for the bars that represent "human score" and those that represent "excellent performance" - so that someone doesn't skim and assume they are the same thing, given that currently both are represented as red dotted lines but mean two different things.
I believe the time has come for an independent institute to track AI and Big Data technology applications and begin creating guidelines for both industry self-certification and regulation. The other paths available, ignoring it, fearmongering about it, and trying to fit it into other political movements, do not seem to me to be heading towards an acceptable outcome.
It has to be a technology-heavy group, otherwise it won't create much value. It also has to be grounded in history, philosophy, and political science, otherwise it'll just be reactionary. And we have enough reactionary groups already.
Nice that it's open and shared, but kinda weird that they are storing what is essentially tabular data in code. Looks to me like it would be much more readable if the data were in csv files, and just the graph generating code were in, well, code.
