I recently faced this same challenge and solved it using Amazon SQS, a simple Lambda function, and a corresponding SQS consumer/relay service running inside the private LAN.
Essentially GitHub's webhook is configured to hit the Lambda function, which authenticates the request and adds the data to an SQS queue. The internal relay service watches that queue and passes requests to the private Jenkins server.
The architecture is similar to this[1] project, but since the public-facing piece runs on Lambda, it costs almost nothing to run.
This is almost what we do as well. The difference being that we have a API Gateway in front that just invokes the lambda on an internal network, which validates the webhook data and only then does it forward it.
Takes the complexity of having to use a queue out of the equation, though at the expense of potentially lost webhook calls.
Looks nice. The key difference seems to be the OpenZiti solution enables you to manage by identities (instead of IP addresses), and close all the inbound firewall ports? However, believe the two solutions would work together?
Yes, if you also need users to log into Jenkins from outside the private network (without a VPN), it sounds like OpenZiti would be a good option. In my case, Jenkins is only used from within the LAN. The SQS solution authenticates GitHub webhooks using the sha256 hmac signature (not by IP), and no inbound ports need to be open.
You'll still have open ports on the LAN. With OpenZiti you can even shut down the host firewall from having any open ports which I think is pretty cool. :) Plus you'll be able to access Jenkins from anywhere at that point - even from home - just like you were on the LAN. Maybe you'll find that useful someday in the future and give OpenZiti a try! :)
Also embracing OpenZiti allows you to give users access to Jenkins securely (or any other app you want/need to keep off the public internet) without using a classic style VPN.
Even if you nail the configuration and have nothing but sweet love for your favorite VPN it's still a perimeter security model and there's no real assurance that only your friends are inside that perimeter. I'll bet you that's not always the case. I don't mean to be ominous but it strikes me as a false sense of security. Going full zero trust is a more meaningful assurance that only authorized devices and apps can connect.
I was moving towards using userspace ZeroTier (via the Python bindings as a SSH proxy command) to allow Bitbucket build runners to access our private network repo server, for a similar result. The bitbucket runners run containers in such a way that the normal Zerotier wouldn't run (something about not having a capability flag for tun/tap IIRC).
In the end bitbucket released self-hosted runners and we're now running the builds in our network with normal firewall rules limiting access.
I'm still looking at overlay networks (probably Nebula), and want to take another look at OpenZiti (I peeked at it a couple months ago).
Author here, I want to learn more about ZeroTier too. I overlooked it for this project because it supports layer 2 and I needed tighter coupling of application+policy than layer 2 provides. Admittedly, I don't yet know enough about ZeroTier to know if I can disqualify all layer 2 points of contact with a particular policy. It sounds like the layer 2 DNA of ZeroTier tripped you up in this case too.
I know more about OpenZiti than Nebula and will be listening for topics in the Discourse in case you decide to take another stab at it. Nebula is definitely interesting. I got the impression that it's all about automating spin up and tear down of point-to-point VPN tunnels on demand, which has the potential to eliminate much of the headaches around maintaining VPNs. Essentially app and policy configuration driven over hand crafting. That's definitely the approach of OpenZiti too.
I haven't really used the layer 2 functionality of ZeroTier, I know it has it. I believe in the built-in firewalling rules in ZeroTier you can limit the protocols that it will pass. The rules for it are, in my limited use, pretty obscure, but I have had some luck at setting up rules.
When taking another look, you may find this brief set of 'superpowers' useful - https://www.youtube.com/playlist?list=PLMUj_5fklasKF1oisSSuL.... This includes closing all inbound ports (or FW rules) so that the apps become dark. I believe ZitiTV is going to have a comparison episode too soon incl. ZeroTier and Nebula.
As someone working on OpenZiti itself, I'd be interested in why choose Nebula over OpenZiti. I assume it'd just be due to the cache that Nebula has, since it comes from Slack? I am biased, I know, but OpenZiti offers a lot of interesting stuff over Nebula.
If it is a usability issue I'd really like to know since I'd want to make that experience better.
I remember my first look at OpenZiti now. Here's some feedback:
My first impression of OpenZiti was: Nebula but at the application level.
It took a fair bit of looking around for me to see the proxy component. I'm still not exactly sure how to get a test, but it looks like the getting started steps are:
- Build a network.
- Figure out the identities and services.
- Set up a proxy.
- Test out this proxy.
- Eventually migrate services from proxy to SDK.
There's a lot of stuff going on here I really need to wrap my mind around before I can even see how it would fit into my network.
Nebula on the other hand is an easier mapping for what I already have in my brain: It's a mesh VPN with a CA and built in firewall rules based on certificate roles.
OpenZiti looks very interesting now, but it honestly took me an hour to get to that point.
Yikes an hour??? I need to do a better job at explaining it then! Thanks very much for taking the time to provide feedback. Figuring out identities vs IP addresses - yeah I can see how that might be a bit of a leap. Makes me want to produce a video/doc on that topic.
"Nebula at the application level" is not far off to be honest. There's probably a fair number of things OpenZiti provides that Nebula isn't trying to provide but it's a decent one-liner type of summary. OpenZiti also seems to cover many/most of the things Nebula set out to do so it's not just for applications.
If you have any requests that would help you get started, I'm happy to try to satisfy them! :)
I have a few questions off the top of my head if you have any thoughts on them. Thanks so much for your replies on this comment thread.
- What is the performance of OpenZiti like? I'm seeing some Nebula performance numbers (from third parties, not clocked myself) that seem surprisingly bad, but I'm also not particularly performance sensitive as far as throughput for most things.
- I predict problems with DNS on 127.0.0.1:53 on some hosts, thinking it might be better on 127.0.0.53:53 though I feel like that might already be commonly used for something. I know on my work laptop I have dnsmasq running there that forwards DNS around, that's the primary place I'm expecting problems, I mean with the exception of my internal DNS servers.
- Automated enrollment? Most of my dev/stg machines respin every other day. I understand enrollment is done via a code that can be e-mailed, so I presume it's a one-time-use code. I need some way for the machines to re-join the OpenZiti after being deleted, a base Ubuntu installed, and then Ansible run. For Nebula this would be by delivering the certificate to the machine.
.53 is the default for the built-in nameserver in `systemd-resolved`, but there's not collision with the OpenZiti tunneler for Linux because it implements systemd's API and creates a tun device and nameserver in a configurable IP range for which the default is 100.64/10.
- The project is actually currently in the middle of some performance comparison testing. I myself am not conducting the tests and I'd like to review the methodology we use for the tests but initial reports I'm hearing are quite good. I do have first-hand anecdotal evidence from a heavy user of OpenZiti indicating from Australia to the US he'll usually pull 40-50mbps over open internet. OpenZiti often pulls 50 consistently while still being a zero trust overlay. One of the knocks on OpenZiti is the mTLS between links, and full end to end encryption on by default being "overkill". This anecdotal evidence however, seems to suggest that the CPU burden of "all that encryption" is not significant at all.
- The private, authenticated DNS provided by the tunnellers has not been a problem whatsoever for nearly every user. Sure there are times when bugs creep in, but I'd say for the majority of times people use any of the tunneling apps the way we do private DNS has been no issue. The OP of this article might have more to offer here. He's very well versed with DNS in general. I don't recall exactly how this works in linux but we have some advanced linux users doing "interesting" things and I know of no major issues... yet... :) If you try OpenZiti, I hope you won't be the first! Each OS is a little different here too. The Windows clients work differently and the MacOS ones a bit differently. All "somewhat" the same but sometimes with subtle nuances
- "For Nebula this would be by delivering the certificate to the machine" - realistically that's exactly the same process with OpenZiti's "regular/most common" type of enrollment - the "one time token" approach (using jwts). You deliver a jwt however you like with email being just one option. Then you just re-enroll that jwt which creates a primary key and cert right then/there on that machine, and off you go...
You might find another feature interesting though, OpenZiti also supports 3rd party CA enrollment. This means you can register a CA with OpenZiti and if any device connects and presents a certificate signed by YOUR CA (you don't even have to trust OpenZiti, you can run your own PKI) then you can set that device up for "automatic enrollment" which is truly automatic. You don't need to deliver a new certificate (but you could). This is a somewhat detailed topic and I don't think I could fit enough words on this post and keep it legible. Would be happy to continue that conversation here or in the OpenZiti discourse forum.
One other interesting thing you might like... We just released our first release of a python sdk for OpenZiti. It's able to monkey patch python to allow us to use ansible over OpenZiti. So hopefully soon we should be able to demonstrate ansible over OpenZiti. I've seen it working but it's not __quite__ ready to share to the world. Soon, I'm sure!
Preliminary project throughput comparison testing in OCI
Ziti Tests were run with a single fabric router running in same VCN with either the source or destination ziti endpoints
Test method: Iperf3 tcp test with default settings with -t set to 60
Competing technologies were tested with defaults as per published installation instructions
Investigating as to whether or not stun/direct connect is working for non-ziti technologies and possibly following suboptimal paths”
Source Destination Ziti Edge Tunnel Ziti Router Tailscale Netmaker Wireguard Zero tier Open Internet
OCP Singapore OCP Mumbai 90.9 Mb/s 127 Mb/s 48.6 Mb/s 28.4 Mb/s 23.2 Mb/s 25.5 Mb/s 377Mb/s
OCP Phoenix OCP Ashburn 95.6 Mb/s 216 Mb/s 57.7 Mb/s 37.5 Mb/s 35.8 Mb/s 18.5 Mb/s 384 Mb/s
OCP Singapore OCP Phoenix 74.7 Mb/s 105 Mb/s 15 Mb/s 16.0 Mb/s 17.7 Mb/s 13.1 Mb/s 115 Mb/s
I'm pretty stuck WRT doing a test setup. My primary issue is getting to the "Run ziti-tunnel run ziti.json", in either the OpenZiti instructions or the NetFoundary instructions, but having no clue what ziti.json contents need to be.
I tried using the jwt I got from OpenFondary and doing "ziti-edge-tunnel enroll -j seans-laptop.jwt", but I got no output, just a 255 exit code:
Gah. I need to file an issue to provide better help here. Both openziti and NetFoundry instructions should be just fine. Your command is just missing the resultant identity file... the -i flag and produces NO helpful output :(
Below I made two identities and enrolled one with the 'ziti cli' and the other with 'ziti-edge-tunnel'
### create identity to enroll with ziti edge enroll
clint_private_aws: ~$ ziti edge create identity user clintaws1 -o clintaws1.jwt
New identity clintaws1 created with id: g1aO5i.7Ig
Enrollment expires at 2022-05-26T21:18:40.289Z
### create identity to enroll with ziti-edge-tunnel
clint_private_aws: ~$ ziti edge create identity user clintaws2 -o clintaws2.jwt
New identity clintaws2 created with id: qCFO5s.YIg
Enrollment expires at 2022-05-26T21:18:42.419Z
Use "ziti edge create [command] --help" for more information about a command.
### enroll with ziti edge enroll
clint_private_aws: ~$ ziti edge enroll clintaws1.jwt
INFO generating 4096 bit RSA key
INFO enrolled successfully. identity file written to: clintaws1.json
### enroll with ziti edge ziti-edge-tunnel
clint_private_aws: ~$ ./ziti-edge-tunnel enroll -j clintaws2.jwt -i clintaws2.json
[ 0.000] INFO ziti-sdk:ziti_enroll.c:94 ziti_enroll() Ziti C SDK version 0.26.29 @cc792e2(HEAD) starting enrollment at (2022-05-26T13:01:17.145)
[ 0.000] INFO ziti-sdk:ziti_ctrl.c:371 ziti_ctrl_init() ctrl[ec2-18-188-201-183.us-east-2.compute.amazonaws.com] ziti controller client initialized
[ 0.022] INFO ziti-sdk:ziti_enroll.c:41 verify_controller_jwt() verifying JWT signature
[ 0.027] INFO ziti-sdk:ziti_ctrl.c:371 ziti_ctrl_init() ctrl[ec2-18-188-201-183.us-east-2.compute.amazonaws.com] ziti controller client initialized
Thanks for that pointer, I was able to enroll my Linux machine with: "ziti-edge-tunnel enroll -j my.jwt -i my.json" and then get connected using "ziti-edge-tunnel run -i my.json" (note that "ziti-edge-tunnel run my.json" doesn't fail, but also doesn't seem to register with the NetFoundry servers).
So that "enroll" step seems to be missing from the various "getting started" guides.
I went through the rest of the hello world example, but ran into various problems. My Mac never showed the service I created. I was never able to do DNS resolution, probably because systemd-resolved is running (and relied upon) on port 53 on the Linux side. Not sure why the Mac side couldn't do DNS resolution, probably similar (but I'm using the Mac DNS services, also relied upon to get DNS resolution to my work VPNs), possibly related to the lack of the service showing up in the Mac Ziti client.
I tried both a service by name and a service by IP, but wasn't able to connect to the IP from either machine, and iptables on the Linux side never showed the IP I gave the service.
I like the promise of OpenZiti, but man is it hard to set up. I'd say I've easily got more time in setting up OpenZiti and still haven't been able to successfully pass traffice, than I have in setting up ZeroTier, Nebula, and Wireguard combined. Just for a point of reference.
> I like the promise of OpenZiti, but man is it hard to set up.
clearly that's not an acceptable user experience! I will take this to heart and try to make a much more clear guide on this. Probably one which does some kind of matrix of all the tunneling clients.
Thank you so very much for your time and energy here. If I can help at all, I'd be eager to.
Your comments are hard to hear, but so very much appreciated! It'll help us make a better overall experience
I can't recall where exactly I saw OpenZiti, it was probably from some discussion on HN. I went to look at it and thought it looked promising, but I'm already partway down the path of Nebula, and as you say it's partly because of Nebula having Slack experience behind it, partly because I have some experience with it, partly I like the architecture. I'll take a peek at OpenZiti again and refresh my understanding of it.
We wanted to stop external network attacks on our internal development environment while allowing tools like Jenkins to connect to other internet based tools such as GitHub. We do not want to have any inbound ports or deal with VPNs, ACLs, bastions, complex firewall rules etc. We dogfooded our opensource technology, OpenZiti to achieve this. We would love to know your thoughts.
>Everyone that needs to log in to Jenkins runs an OpenZiti tunneler on their computer. Each tunneler has an identity installed on the same computer. The OpenZiti network administrator issues these identities via email or chat in the form of one-time enrollment tokens and assigns role attributes that match up with a service policy. The service policies grant permission for those tunneler identities to use Jenkins. We control the service policies through a web console or CLI. As soon as the policy grants access to an enrolled identity, the tunneler automatically configures a private domain name and IP route to the Jenkins server on that user’s computer.
How do you get everyone to run the tunneler on all the machines that they could use?
For example our company uses subcontractor companies that have their own network admin team that manages their work computer and a strict VPN. Getting an employee to run a tunneler sounds like a non starter.
> How do you get everyone to run the tunneler on all the machines that they could use?
There's apps in the stores for Android/iOS/MacOS and installers for Windows. It's basically equivalent to running any other "VPN type of client". Wireguard does the same thing (in fact for Windows we used Wintun from Wireguard which was awesome). OpenVPN/Cisco/ZeroTier etc.
OpenZiti is also working on an "install-less" option for users that can't (or don't want to) install apps on their own. It's not ready/rolled out but hopefully it will be soon.
The tunneling apps provide some pretty cool functionality though, most notable is private, authenticated DNS. IF you were running the tunneller you could go to "jenkins.ziti" or whatever made up DNS name you like.
If you can't install a vpn client like that then yah, it'd be a non-starter for accessing jenkins like this.
You're absolutely right. Its an interesting solution to the problem but I'm not sure what additional benefit this solution bought them that isn't covered by standard paradigm of VPN w/ role based access controls. (at least for user access)
They did however correctly identify the fact Jenkins is pretty much the holy grail of targets for bad actors and hackers. Failing to identify this has caused more then fair share of known hacks (to not Jenkins specifically but any build/automation system that has the required insane levels of access).
A strict VPN locked down and for external ingress access of automated actions a restrictive proxy sitting on the edge significantly lowers the attack surface. Operationally also much cheaper to maintain.
This speaks to a strong nuance, its not only the closed inbound ports meaning no external network attacks, the strong identity allow the user to be anywhere (home, coffe shop, holiday) rather than tied to an IP while Sec team also get massively visibility of exact who/which identity (rather than IP) is accessing what and when. Its a stronger security, better visibility, greater velocity with higher automation.
can be difficult but one key is the openziti tunneler is the opposite of a traditional vpn client - it only intercepts apps by policy, e.g. it would protect the Jenkins traffic, but ignore whatever else the contractor is doing (assuming that is the policy).
there are also gateway models (contractor to gateway leg is left alone; the openziti magic then applies between gateway and Jenkins/your enviro).
are there other good options for this contractor scenario?
That's pretty nice, but won't work integrations that don't let you run custom code.
In those situations it's a good idea to run an authenticating reverse proxy in front of your Jenkins instance, and allow certain paths in case certain query parameters are set.
So for regular access, your users need to auth with your SSO.
Webhooks include something like `?apiToken=12345` in their request, which you check in the reverse proxy config, and then use to allow access to the relevant URLs.
To your point, not all SaaS or vendors with integrations let you run your own code. If they do allow it there's probably an OpenZiti-native way to build-in the connectivity you need.
One alternative is running a tunneler like our users did for their continued access to the Jenkins UI. This is like a VPN client that can run pretty much anywhere, and so there's an excellent chance that it's already in the app store or coming soon for whatever platform you have in mind.
As for an authenticated proxy, thanks for that idea. I do think that a hardened internet edge like that has utility in cases like this, like NGrok + SSO. Another reader suggested their solution involving a third-party queue, essentially offloading the internet exposure problem to a public cloud provider, which is basically what I did here but with different software.
It definitely was *the way* to do it before OpenZiti and other zero trust overlay networks started to take off. Being "in the VPN" is similar to participating on the OpenZiti overlay network. VPNs allow for horizontal movement much easier than being on an OpenZiti zero trust overlay network does. With OpenZiti you don't need any holes in your firewall open. That's a pretty good reason to consider moving away from VPNs to a zero trust overlay like OpenZiti in my book, but I'm biased, I work on the project. Embracing the zero trust networking mindset is definitely a change of pace but I think it's worth considering.
Good job avoiding public internet exposure, but the problem with being "on the VPN" is that it has all the problems of the internet at a smaller scale, so you're still exposed to an untrustworthy network. I'd say that's the core tenant of so-called "zero trust" philosophy.
In the past I have just restricted Jenkins access to our main static IP, and whitelisted GitHub's published Webhook IP ranges. Remote employees just need to login to the office VPN to access the UI & SSH.
Does the articles method provide further flexibility? It seems much more complex!
It's probably "similarly complex" but it's more complex at start because you won't have an OpenZiti overlay network already... So that's more complex for starters. Once you are familiar with OpenZiti it'll be easy. So there's an uphill swim for you since you're already familiar with SSH and firewall rules...
OpenZiti's big advantage here is that if those users pick up and move to starbucks, dunkin' or wherever - you don't need to whitelist an IP any more. it'll all "just work" because OpenZiti treats all networks as hostile/compromised following the zero trust networking principles. There's also a LOT of other stuff that OpenZiti will provide that is above and beyond SSH.
One other big factor is no open holes/firewall rules. That's a big benefit imo too.
So yes - there's a lot more flexibility with OpenZiti vs SSH/firewall rules. I don't want to just keep going on and on here (but i can :) )
I think the flexibility of operating independently of IP addresses is the key distinction of this approach. It also works for the web UI, not just the webhooks from GitHub.
Essentially GitHub's webhook is configured to hit the Lambda function, which authenticates the request and adds the data to an SQS queue. The internal relay service watches that queue and passes requests to the private Jenkins server.
The architecture is similar to this[1] project, but since the public-facing piece runs on Lambda, it costs almost nothing to run.
[1] https://github.com/osbuild/webhook-relay