Author here: we did? Pretty well documented issues w JSON and YAML (coming from CloudFormation perspective). Both have problems (that are unique to each) so we created a format for solving that.
If you like JSON and YAML and don't mind those issues you are well served by existing solutions. Cheers!
What problems did you solve and how were they solved with your new format? I'm neutral in this argument at this point, so I'd like to hear your justifications. Thanks!
- JSON has no comments
- YAML nesting tends to get deep which makes creating bugs very easy
- Both are become hard to read / edit in larger files
The primary issue is translating the intent into a machine syntax. The format we landed focused on minimalism to communicate intent.
The the file format is the least interesting thing about .arc but one of the most distinctive. An .arc file VERY QUICKLY communicates the project intent.
An example:
```
@app
hi
@html
get /
```
The equiv CloudFormation YAML is 700 lines. You'd have no idea what that project does without digging into the code. The .arc by contrast is extremely easy to both read and author. As the app grows it becomes very clear the project intent and were to find things.
We certainly could have created a new YAML thing but many projects already do that … and I'll let their work speak for itself.
I am not sure, but serverless requires more config and tweaking as compared to arhitect.
In contrast, serverless is a more mature and flexible platform.
YAML works pretty well for the 80% case, I think. I work at Pivotal -- we use YAML so widely in our designs that a few years ago there was a swag t-shirt with the title "Enterprise YAML Architect" printed.
YAML's shortcomings can be addressed a number of ways. The most recent trends I've seen are:
1. Simplify the YAML itself.
This usually means introducing higher-order features that make long, repetitive blocks irrelevant. An example is BOSH v2, which introduced a number of features (Eg. BOSH links) that pushed repetitive scutwork into the tool, away from the YAML manifests.
A variant of this is to use more advanced YAMLisms to make your YAML files more concise. Anchors make a big difference. I've seen this be very useful in large Concourse pipelines.
2. Use something that renders down to YAML.
For example, Jsonnet[0] or Enaml[1]. Some folks will use templating languages for this purpose (erb, mustache etc).
Templating is a last resort, in my view, because it's hard to see what the output will be without rendering it. Plus most tools format nested syntaxes poorly.
On the upside, use of a full templating engine introduces opportunities for multi-file DRYing. But that itself should be a smell that the configured tool is missing some feature or capability.
3. Use a different format.
TOML appears to be the most popular alternative to YAML and JSON. Most recently I have seen the joint Heroku-Pivotal effort on Buildpacks v3 adopting TOML over YAML or JSON.
I'd like to leave the format discussion behind. Its the least interesting part and you rarely interact with it. I'll add YAML and JSON support so ppl can judge for themselves on merit against options that have tradeoffs.
Its absolutely suitable today. We have it in production in several apps. begin.com and arc.codes itself are two examples that have seen non-trivial traffic.
One of the biggest promises that I can see about the serverless movement is:
The ability to have the infrastructure isolated from the raw code being written. (Think more of a grid computing and automatic planning being included)
Unfortunately, I'm not seeing it. I'm seeing that it's heavy on the initial business idea: "Get people to pay more and utilize our servers better".
> The ability to have the infrastructure isolated from the raw code being written. (Think more of a grid computing and automatic planning being included)
Can you elaborate? It sounds like you're saying that if you write code that's deployed to a serverless infra you end up with 'serverless code'. This hasn't been my (very limited) experience - where the serverless stuff is just a thin layer around your business logic.
I think they’re saying that you can expose your lower-level atomic APIs as serverless functions, and then, separately, build a “routing and aggregation layer” that exposes APIs that people might actually want to use. And that, for a lot of projects, that routing/aggregation can be entirely specified in a declarative manifest format, rather than involving any coding.
This neatly cleaves responsibility: your business logic team can just build+deploy+expose functions; and your frontend and ecosystem teams can build+deploy+expose the APIs they want on top, perhaps being maintained within their own codebases.
These can even be done on two entirely-separate platforms, rather than both being a part of the same “serverless” IaaS. You could use any combination of commodity FaaS providers (including your own hosted FaaS clusters) to deploy functions to; and then you could use some “smart” load-balancing system (e.g. Cloudflare Workers, or the routing component of an FaaS) to do the routing+aggregation on.
DISCLAIMER: I've contributed a few times to the Serverless framework.
I like some things about Architect, but there are things that just don't make sense:
- Integrations with Slack but no way to define custom CFN resources
- No authentication support
- Custom file format (3)
- Not being able to set CORS in this custom file format (4)
- Magical vendor folders (5)
I am sorry for the negativity, but brianleroux has posted in this thread several times claiming his framework saves 700-line CloudFormation templates. What I don't see is him comparing snippets with say, the Serverless framework(1), or AWS SAM(2). He seems to portray the choices as either his pet-framework-of-the-month or plain old CloudFormation, and that's just not true. That's not to say Serverless or AWS SAM are perfect (they most certainly have their warts), but they are both arguably more mature and have larger communities behind them.
Hey np! Difference of views. No integration w CF is absolutely true but you're also totally free to do that. It doesn't prevent you from using it.
No auth support is false. We support session out of the box and you're free to impl any auth scheme of your choosing.
Custom file format is being well addressed in other comments. If you don't like it thats fine: enjoy YAML elsewhere.
CORS could be implemented in the file format…I'm personally not a fan for a variety of reasons but its open source. PR's welcome.
Magic folders! Absolutely. Structure is a good thing.
I don't find this negative. Serverless framework is an abstraction that is easily as verbose as CF but I'm not interested in comparing to it because we have different goals. Serverless is an abstraction between clouds. If mulitcloud is your goal: use Serverless. SAM is a flavor of CF which is def less verbose but still very vendor specific and …wordy.
This isn't a 'pet framework of the month'. I do take mild offense to that. Development of .arc started in 2015 for begin.com and got met with some serious production traffic (from things like Techcrunch and Venturebeat). It held up beautifully. We decided to pivot from our original producton into CI/CD for arc with begin.com because it is clear to us that it is a better way to build serverless apps.
Architect was officially open sourced in July of 2017 and donated to the JS Foundation to demonstrate the long term commitment to it. Development has been very active since. I stand behind my work, dedication to open governance and approach we're taking and, I think, I have the track record to ask for the benefit of the doubt when I say that.
> No integration w CF is absolutely true but you're also totally free to do that.
It just seems then that there is no real way for me as a developer to transfer all the accumulated knowledge I already have about AWS to your framework. Looking through Architect's source, it actually seems as if everything is done via the AWS Javascript SDK, which seems a bit ... odd, to say the least. I'd even venture as far as saying it goes against the AWS Well Architected Framework guidelines (1) (which recommends using CloudFormation).
> No auth support is false. We support session out of the box and you're free to impl any auth scheme of your choosing.
Apologies, I couldn't find any mention in the docs. I meant more in terms of using an AWS service such as Cognito Federated Identities, and in terms of protecting API routes with either AWS_IAM, API keys or Cognito Authorizers.
> CORS could be implemented in the file format....
An Architect project deployment is not exactly repeatable if I still have to go to the Web Console to enable CORS now, is it?
> Magic folders!
I personally would prefer either my project's root `node_modules` or a Webpack bundle be deployed, over which I have some measure of control. Node already has a hard enough time with its resolution algorithms, must we add to the madness?
> Serverless framework is an abstraction that is easily as verbose as CF
I can't agree with you on Serverless being as verbose as CloudFormation, and I think you should know better as someone who has dealt with the complexity of setting up API Gateway programatically.
> but I'm not interested in comparing to it because we have different goals.
I think the comparisons are fair - both Architect and Serverless claim to be "Serverless frameworks". I could be in the minority though.
> Serverless is an abstraction between clouds. If mulitcloud is your goal: use Serverless.
I mean... you're not wrong, but that's grossly oversimplifying it. I've personally added support for Serverless AWS projects to allow hooking up Lambdas to Cognito User Pool triggers. That's a pretty solid AWS-only use-case, that likely isn't available on other cloud platforms (or at least, I never implemented it for them).
> SAM is a flavor of CF which is def less verbose but still very vendor specific and …wordy.
Well, isn't Architect vendor-specific as well?
> This isn't a 'pet framework of the month'. I do take mild offense to that.
Apologies for that remark. I'll be honest - I'm not really interested in the story behind the framework, I'm more interested in what it brings to the table. I'm glad your project worked for you and your company, and I wish it the best, but personally I don't see why Architect couldn't have just been a layer of syntax sugar over SAM/Serverless framework. If there was a killer feature that's specific to Architect, I can't see it (unless you count the file format as a feature).
Long story short is this existed well before CF support did for Lambda. I don't think you're actually interested in learning and more interested in appearing to know what you're talking about so I'll leave it at that.
Also! Killer features galore but folks having trouble seeing past the fmt. You're just making so many assumptions here it's very clear you didn't try it. It's fast.. really fast. Faster than CF.
I actually like this. I build serverless apps all the time, mostly manually.
Serverless framework is alright, but I felt it was really bloaty. There was just so much going on. Too many variables and it confuses the user. This looks clean. I might give it a try. I didn't find a link though in your blog.
There is a link, but it's buried in the middle of the second paragraph. There should really be another link in the call-to-action section at the bottom.
I've wanted to try building something as a serverless architecture for a while, but every time I look into it I get confused by all the different things you have to set up to get an AWS Lambda function working, or I look into a framework but I can only find pre-canned examples for hooking into Slack, or just generally get confused by blog posts with lots of buzzwords and very little substance. In the end I always wind up just writing another daemon and sticking it on my server to get the job done. All I want to do is write some code and trigger it with an HTTP request, but for some reason Architect is the first framework I've seen that puts an example for how to do that on the homepage.
Plug that I’m sure Brian won’t mind (he encouraged me to start a company a couple years ago):
If you want to get started building serverless functions and APIs quickly, do it in your browser with https://code.xyz - we just released it along with an announcement of our most recent $2M in funding from Stripe [1] who join Slack and others as investors and customers.
See, this is exactly the sort of thing I'm complaining about.
code.xyz gives some 'hello world' examples, but it doesn't even link to any docs--it just goes "look how easy it is! Here, sign in to our service!" I went hunting for the stdlib docs, and it took me five minutes to figure out how to call my service by HTTP--apparently I just get a special username.stdlib.io endpoint that I can call, but when I clicked 'Calling Services' the first thing I got was a page that explained how to use bearer tokens. I still don't know how I can POST to this endpoint, but I have learned that apparently stdlib has invented the concept of a 'source code' because apparently having a pre-canned Slack bot integration is super-important to any self-respecting FaaS provider. Also apparently I can find prebuilt 'source codes' in the 'stdlib registry', but the docs don't link to said registry, only the aforementioned Slack integration. At this point I gave up; I don't have the patience to learn a brand-new meaning for 'source codes' when all I want to do is make an API call when I get a webhook payload, which I already know how to do in a dozen lines of Python.
I realize this complaint sounds kind of harsh; you're getting a lot of my frustration dumped on you. Really all I want is basically a managed, autoscaled version of Express.js. Maybe there's some reason it's like this (I've never worked with anything that could be remotely described as 'at scale', it's mostly been line-of-business apps with a few dozen users maximum, or at most a web shop that still runs on a single dedicated server), but it seems like all of the Serverless providers add a bunch of extra concepts and confusion on top of what should really be a simple principle. Or maybe I'm completely misunderstanding the point of Serverless and it's actually intended to solve a completely different problem I've never had--I'm also open to that explanation instead!
When you run your code a link to your API endpoint is provided immediately: obviously this isn’t clear enough! Thanks for the feedback. We’ll make sure we fix this ASAP. :)
I’m a big fan of code generation. I can usually adapt to new formats or concepts without too much issue. I feel that while it’s not difficult to build up strings dynamically, the harder part can be overwriting and verifying values, and working with logic. That’s my main issue with YAML over JSON. How would you recommend working in this area with this format, or benefits over JSON?
YAML is far easier for humans, and has a few advanced features (node references) that are REALLY nice for any kind of configuration, since you can re-use blocks.
Let's say you have to define a bunch of services, with mostly the same environment variables. You can make a "defaults" node, and then a) reference that node in the service descriptions to re-use those env vars, and b) selectively override those defaults in the service description's node while leaving the rest of the defaults untouched. It's really nice.
It's super useful. There's a way to reference a node that just places that node as a child, whole cloth, and another way that merges the node reference with the local node.
This is useful, say, if you're actually de/serializing objects and want to maintain actual references, or if you're just trying to share data between nodes.
I'm not the creator of this project, but JSON just isn't fun to write. It's readable, but still icky to write. I think to create a format really needs to strike the balance between usability and how natural it is to write. I think ELM for example finds a good balance.
It can be very expensive. I priced out two scenarios recently for an IoT solution. Using Lambda the cost would be $27,000k a month. Using a LB/EC2 solution with reserved instances the price was $60/month. (We also priced out IoT core, and it was $3,000k a month).
Can you share more details on this? We've just moved to Lambda and are paying approx $60/month for lambda where we were paying $260 on heroku.
But then our db on AWS is another $110, where it's included in heroku.
I know it will depend on your memory usage, etc. and as we grow, we expect to move to ec2, but I'm curious as to where/how you had such a huge difference in cost.
I really wish people wouldn't use JSON for configuration. The lack of comments and shortcomings in formatting abilities make it particularly ill-suited. { "_comment": "this is a comment" } is cringeworthy to. It's great for transport, but horrible for static file config.
YAML is good, but not to everyone's taste. I've been using it for a long time (as a ruby developer) but many people I know find it ugly.
I think Arc config looks much better than anything they could have come up with using JSON/YAML. Optimize configuration files for humans, not machines. If its simple enough to learn and be readable, more power to em.
It's interesting. can you have @js more than once? can you have @app more than once? pretty sure ಠ-ಠ is a valid app name, but ಠ_ಠ isn't.
i also can't quite tell how to say map of maps. i'd guess it's
@this-section-has-a-map
hello-world
name child-map
ಠ-ಠ a
ಠ_ಠ b
but it's hard to tell.
Also, i'm not exactly a fan of { "_comment": "this is a comment" }, but i've run into several situations with yaml where i want to build a tool to rewrite a bunch at once, but since comments have no representation in the parse tree i have to do goofy vim macros to do my edits. since _comment is in the tree, i can add more keys without trashing the existing comments.
Hey, one of the .arc authors and maintainers here.
Only one of each section is what we designed for though the validations for that might be broken (and if so we can fix right away). Nesting is limited to one level. No need for more that we've seen yet.
I'm a proponent for JSON but agree that the lack of comments are one of the more annoying things I frequently run into.
edit--Especially when it comes to users editing config files. Imagine a new user trying to edit a web server/db/.bashrc file that didn't have comments to help out?
Rather than inline-comments I've always had separate documentation. This helps prevent trying to cram all the edge case caveats and everything into the configuration file.
inline comments are handy for pointing out "non standard" configuration settings for your enviroment. especially on production systems.
It aids in troubleshooting greatly to have inline comments point out "odd" settings in configuration without having to look at seperate (usually old) documentation.
Then use json5. Super weak excuse. Or use yaml or some other format and pipe it into some to-json converter.
Also, json has json schema which tools like VS Code support. They can autocomplete your config. Pretty promising over most other tooling.
But coming up with your own config has pretty much all the downsides. Yeah, you can invent some cuter syntax. But now your users can't even get syntax highlighting in their editor. Awful UX.
JSON5's support for unquoted keys seems like a misfeature. Not to mention that (by its nature) it has the same (lack of) support burden as any other new configuration format. If you're going to go through that frustration and incompatibility, why not pick something better suited to the job?
https://serverless.com/ - uses yaml or JSON or js. This looks like an opportunity for the community to grow some more which is almost always a benefit. A new filetype isn't always the best, but it CAN be useful in some contexts, I haven't vetted if this is the right one.
Yup, serverless works quite well. have used it myself.
Albeit, I'm not fully sold on this whole fake-serverless architecture.
A major part of it is that AWS is arguably frustrating, since the docs are __clearly__ written by marketing people, not engineers. Every aspect feels non-dev friendly.
I would love to see something like Captain Duck Duck, so you can host your own serverless platfrom, sort-of a personalized AWS?
I don’t think you get my gist about syntax vs information. I can express the same data in a different format, in the same way that JSON and YAML can often be transformed into each other.
You are pulling a straw man by implying that requires much cloudformation, which is completely unrelated. Besides, CF is a good example of how you can use two different syntax to describe the same information.
The format of a configuration file has no relation to the output of the program that parses it. Why are you being so obtuse about this? Your three-line arc file would be a three line YAML file. Stop doubling down on this nonsense.
My problem with the serverless
movement is database connection pools.
Every time a function runs, it has to create a new connection to the Postgres database so it can run a query. This is super slow, and dead in the water. If I have to maintain a separate server to have a global connection pool, I might as well just have a server for my API as well.
I agree with this. I would be comfortable moving more code to serverless if this weren't as big an issue. I think having a GraphQL server as an API to the database may be a good solution, since it can handle a variety of queries/mutations and RPC like behavior. But I still run into this issue a lot whenever I make a new serverless function/project.
I think this whole idea of pushing "serverless" as a concept to engineers is deceptive at best and damaging at worst when you think about all of the naive new developers entering the industry having to shovel all of this shit into their minds alongside proven technologies and conventional formats.
What does "serverless" framework really tell veterans? Who are you selling to? What are the considerations for adoption?
There's so many questions that pop up before I even look at the software, and once I did, there's more questions which obscure the point. Why should I use your bullshit?
Worse, 5 years in after a company has adopted this junk, some poor sap is going to have to maintain it after being 100k SLOC deep in it, with no vision of how to port out of it alongside meeting existing technical task completions, at which point this will have become completely irrelevant.
> I think this whole idea of pushing "serverless" as a concept to engineers is deceptive at best and damaging at worst…
To me, serverless is as normal a progression of abstraction as anything in software development. What about FaaS crosses a line for you, where (presumably) PaaS doesn't?
