I work in midtown NYC near a Chick-fil-a location. Used to be a line down the block, went with a friend yesterday thinking we caught the line at a short time but turns out they were using smiley employees with ipads coordinating orders and that they were serving arguably more people than before. Very impressed with how seamlessly they've integrated tech in their brick and mortars in contrast with the countless "cashless" restaurants that just add cruft, inefficiency, and customer resentment.
I have a friend who used to be a shift manager at Chick-fil-A. I can't remember exactly what he called that process... It was like meet and greet or walk and talk, but it's surprisingly efficient for the drive thru. He said it cuts down the average wait time for customers around 15% at his location.
I don't know what it would be like at NYC since it's not a drive thru.
It's crazy efficient. When you walk in someone with a tablet takes your order, and by the time you get to the cash register your food is ready. Works so well because the employees are able to catch customers at the end of the line.
Slightly off-topic, but what's the reason that Americans all tend to line-up at the drive thru? I was in LA, there was a huge line at the drive thru, so we parked, went inside and got our orders really quickly.
The comfort of still being in your own space. For example, listening to your own music, no excursion for your kids to belabor, precisely don't have to park and get out.
Yeah, I think I would rather wait 10 minutes in my car than 5 minutes in line. I'm sure other people may feel differently. And something about the drive-thru still "feels" faster, even if it's not. I think it's just the idea that you get to pull out and be on your way after you've gotten your food.
At McDonald's at least the drive through is prioritised over walk ins. That said if the queue is still long it might be quicker to walk in. But inside if orders come at the same time they do the drive through order first.
I guess it also stops people from leaving. Since once you made an order you're more invested to stay and pay for it. But if you were in a long line you might get fed up and go somewhere else (or get a phonecall to do something etc).
True, but I've generally found the line moves fast even when the line is out the door. The main bottleneck in most fast food places are putting your order in at the cash register so if that's removed it speeds things up a lot.
Chick-fil-a does not run the iPad software but it does integrate with their systems. The same company who sells their POS software runs the iPad software and I believe the Chick-fil-a One app.
For sure. Meant it more as a response to contents of the article that they've managed to break down their business processes into a subset of (dare I say) microservices that have allowed them to pinpoint weaknesses in efficiency and improve upon them - ie. the waffle fries example mentioned.
Why do you consider the cook "a poor sod"? Because he/she is working a job and making a living? I imagine you didn't mean much by it but understand that just because everyone isn't employed in what many consider desirable professions doesn't mean their work is not important or that they don't take pride in what they do.
Not meant to be a rant, just wanted to make the point.
I actually interpreted the comment a bit differently. I read it that the employee who used to have some autonomy on when to make adjustments is now doing it because the computer, which they (perhaps) have no control or insight into, is telling them to. I'm sure there's more nuance and people who would view that instruction differently. But to the comments direction, I'd assumed it was alluding to a dystopia of sorts where we do more and more things "because Computer™ said so".
In hindsight, “sod” wasn’t the right term to use (that’s what you get for posting at midnight) but the point I wanted to make was the removal of meaning, autonomy and human interaction when not only are you doing a repetitive job, but how you’re doing it is dictated by a machine. The movie Brazil came to mind, although I’m sure there are better parallels to make.
You'd be amazed how much satisfaction you can get out of learning to keep a steady stream of perfectly-crispy fries going. It's quite fun on a busy day - hectic, but strategic, and rewarding when you can stay on top of things _and_ produce minimal waste in the process.
In some sense, the automation of demand prediction takes some of the fun out of gaining that intuition yourself. On the other hand, it is nice not to be the queue blocker with orders piling up behind you because you weren't prepared for a sudden burst of traffic...
There’s a short-order chef game my kids play on the iPad, you can practically see the endorphins pumping when they manage to serve all the customers in time.
The human brain is definitely wired for this sort of thing.
From my days in those kinds of jobs I'd welcome it. I won't have to go count how much we have on hand. I just drop whats listed and move on to something else. My problems would usually come from "a system" that were missing large steps or prerequisites. "Start frying 1 batch of fries and other in 4 minutes" except the fries weren't cut the night before so both fries wont go in for another 20m. Or the food in the fridge is no longer fresh enough to serve, so we're down some food for the dinner rush.
What!? I LOVED my fast food job as a high schooler. It was fast-paced, and good work. I likely would have enjoyed getting instructions from a computer, lol. I certainly wasn't some poor sod.
That is strange to have a cashless restaurant. If anything cash is preferred. I know of a few places that only accept cash. And in the last week my debit card chip has been malfunctioning so I have to pay cash for everything until my new one arrives. I even attempted to use nfc today. That is a shit technology and was a waste of time. The POS terminals simply don't work.
I got my first taste of real computing from a guy in the next town who would buy dead PDP's from local McDonald's restaurants, fix them up, and sell them on.
He always said the hardest part wasn't replacing broken parts, it was that invariably the machines would always have soda spilled in them, no matter where they were stored. Cleaning the insides was his most time-consuming task.
One of the advantages of an Intel NUC is that it probably has a lot more horsepower than an old PDP/11. It's probably also a smaller target for wayward beverages.
I'd like to hear the story of why a McDonalds would need a PDP. Weren't they...expensive? They would be at the heart of university's IT department, or running a steel mill. Stuff like that. But until PCs came out, at a restaurant I can only picture registers and a dumb terminal hooked up to a distant mainframe.
The logistics improvements from early computers were massive. If you compare the equivalent price nowadays it's not worth it compared to other options, but going from paper to computing wasn't just an incremental improvement but allowed for entirely new abilities.
It wasn't like having an account to crunch all the numbers on staff full time was cheap. Especially when you'd have to have 3-4 accountants just to have 24 hour service, regardless of the work load
I spilled beer all over my laptop keyboard once, so I stripped it down and cleaned down the whole thing with isopropyl alcohol. It was a really annoying task, I can't imagine how annoying it would be with soda in a PDP.
I'm just trying to understand, I see three of those machines, inside a pretty nice rack, with the switches, routers, a keyboard, I would assume some fairly high quality hard drive / RAID type of thing etc. then the blog post claims "We run our Edge infrastructure on commodity hardware that costs us, ballpark, $1000/restaurant. ". Just the intel nuc I can find on newegg alone costs $349 each, with no memory or harddrive installed, retail. I know they are likely getting amazing volume discounts but it still seems amazing that whole stack could possibly be only $1000 each.
Caleb,
I'm working on a project with a similar architecture (on-prem services). I'm curious what your auth-pattern looks like as it's something we've struggled with. Mostly the balance between being convenient and being secure. It seems like each site needs an api key to access your cloud. Do you have an auth-pattern that would prevent the key from being readily available to an attacker who got access to the machine? Or do you kinda just say if they get access to the machine, then it's game over? Also with shoddy networks it is difficult to be confident that key-rotation will happen successfully. Happy to get your thoughts. Cheers.
Just curious (since I work in edge computing products) the deploy here seems rather complicated... What are your feelings on the matter and is there a market for simpler deployments?
I'd assume it's 3x$349 that they're referring to in the article, since most places these days already have some networking equipment (probably including a racked switch someplace) in order to provide guest wifi, connect POS terminals, etc. So that portion isn't directly tied to this effort.
This is Brian -- I wrote the article (I suspect I will be typing that a lot tonight). We do clusters with 3 nodes per restaurant, so at full scale (when we have rolled out to every restaurant in the chain) that will be ~6000 nodes, and growing by new stores * 3 going forward. This will support an estimated 100k IoT "things" of various types in the next year and a half to two years (estimate)
Hey Brian, I have been investigating deploying a very similar stack to what you guys are using now. How are you handling onsite load balancing? Is it a simple round robin type load balancer at the router level?
Also is Highlander open source? I don't see a link to it in the article.
We will open source highlander eventually, but it’s not quite there yet.
We use a VIP that the NUCs share... ie; one of the three will always have a VIP, and if it dies another NUC grabs it. This is a poor man’s load balancer in that sense, because we only have the NUC hardware onsite;
NUCs are surprisingly well-built little machines. I have one in my car, and I give them my absolute seal of approval. I've heard of people running little VMware clusters on them too.
It's actually suprising that people think they're not "normal" computers. I provisioned a few of them and they're little beasts. Most in use are I7, 16-32gb ram, Dual NVME. The Skulltrail nucs are extremely powerfull and on the lower end the atom based units are solid too. I also ran a nuc in my car for a while, mostly stumbling radio / wifi spectrum as I drove around, but it was perfect with its relatively low power requirements.
I have a custom piece of software that powers a display in the car, syncs and plays my music, and logs GPS. It also can manage audio and video recording and is intended to handle some future features[1] I have planned, so I wanted to over-spec rather than under-spec.
[1] It's the same software that runs my home automation system, so an inevitable feature is my car and home units interacting, I wrote half of a piece of navigation software before I stopped, and the entire interface is intended for voice control so I need to throw that in there at some point too. I intend to work on a CAN interface to connect to the car, but I've had a few roadblocks to getting started on that project.
Have you documented any of this or written on your home automation endeavors? I'm looking to "Tony Stark" my house up a bit (i.e. voice control, some touch enabled panes in the kitchen/office) but want a bit more control than something purpose built, off-the-shelf while also being a bit more insulated than Alexa or Echo. The NUC has been on my radar as a device that I may go with as the brains of the operation, and I'm just curious to read what design and implementation patterns others have taken.
I get asked to do more of a "write-up" a lot, but it's hard to figure out how to format or frame it. I have a fairly large pile of opinions about how one should go about doing this, but it's hard to concisely organize my thoughts on it into a coherent write-up.
The code is here: https://github.com/ocdtrekkie/HAController but I don't know if I'd really recommend others use it. The main perk to me is that it's designed around what I want and use (I tried a lot of alternative options before rolling my own), so unless you also really like Visual Basic code and have a brain ordered bizarrely similarly to mine, you may want to start somewhere else.
I'd love to chat about what I've learned along the way or what ideas you have (because I might borrow them!), if you want to chat elsewhere hit me up at inbox (at) jacobweisz (dot) com
I once left the reading lights on; car was supposed to have turned it off when the doors locked, but somehow didn't. I found myself calling AAA in the morning to get a jumpstart.
Also curious, does GP have some way to shutdown gracefully when the car turns off, and reboot fast when it turns on? You could potentially handle this with a laptop and some battery-settings but not aware of an external battery solution for the NUC.
I currently use a laptop power adapter for the cigarette lighter, which is only on when the car is. The NUC's BIOS is set to turn it on when the power comes on, but I power it off manually before turning off the car. This is not ideal.
There is an external battery solution for the NUC, and I bought it! http://www.mini-box.com/NUC-UPS But I haven't switched to it yet. The big upside is that it will gracefully shutdown my NUC after I power off the car. But if I use the cigarette lighter for power, it will lose the ability to turn itself on when power is present, because the batteries are always present. NUC UPS supports using a different power connector which can turn on the PC on ignition, but I haven't had it installed in my car as of yet as that's a little bit more work, and I'd need some help from someone who knows more about the car's electrical system.
As far as fast power-on, it's a pretty high end i5 NUC with an NVMe SSD. It boots pretty darn fast, and my software takes less than a second to load once the OS is up. The slowest part of boot is that I don't want my location history easily steal-able, so it's encrypted, and I have to key in the code to unlock the machine. (I'm looking at a security key or similar to replace this step in the future.)
As a note, I'm more or less specced out what's involved in a solar power setup on the roof of my car to independently power my computer all/most of the time when the car is off and otherwise charge a secondary battery off the car's inverter, but there's no reasonable or sane reason to do it. ...But I thought about it.
Thanks. Curious, where do you stash the NUC in your car? I imagine it's insulated and not exposed -- otherwise a hot day or a very cold day might kill it.
You don't want insulated and hidden, you want perforated and airy (while still being discreet).
The NUC-like hardware that goes into tanks have chassis consisting of a mesh cage to allow maximum airflow since the cabin temperature alone can exceed 100F.
Actually, it's pretty important for the NUC to get good airflow so it doesn't overheat; insulating it would be a bad idea! I haven't had any temperature-based issues with it, in part because most of the time the car is running I am in it, so I keep the temperature at temperatures I am comfortable in. And the NUC's temperature range is pretty wide as well, which is good, because weather here ranges from -10 F to 110 F.
Probably the only real concern I'm afraid of is condensation when heating up the car on a cold day, but it hasn't been an issue so far, perhaps because the computer is not near a window.
Since I'm not a hardware guy, I try to avoid assembling my own gadgetry as much as possible on this project. I use consumer grade home automation modules, standard computer parts where possible. (Everything in my car is largely interconnected with USB.) And generally, experimenting with the NUC UPS aside, I try to avoid parts that may be hard to replace. Which is to say, I don't have time to build, test, and fix a random Arduino gadget. ;)
I've been planning to make some custom-cut USB cables and a 3D printed part just to make the setup of my display a lot cleaner... and I haven't done either of those and it's been a year or so since I planned to.
Nice machines, fast enough, VESA mount to the back of a monitor, and easy to upgrade. They've basically replaced Mac minis and iMacs in our labs. Have not had one bit of trouble with any of them.
they've kind of a pain in the butt for us... For example, it would be nice for them to have remote admin, more than one NIC, not freak out when the HDMI is not plugged in, etc.
Interesting as it seemed that smarter edge/iot nodes would be like ARM/RaspPi, yet it seems it may get even as high as (or just straight to) Intel nuc.
This is Brian -- I wrote the article. For what its worth, we considered trying to run our clusters on an array of cell phones since they would natively support connectivity fallback and pack a real effecient resource punch in a small footprint. One of our engineers came up with that idea. With the challenges compiling on ARM and the relative ease of moving forward with x86, we went that way to start. We tried really hard to balance the "ideal" with the short term "MVP", and I think we landed in a decent spot with the NUCs. They give us enough power to run the things we need in the short term, and we have some capacity for horizontal scale in the future as our needs increase. We tried to think "cloud native" at the edge as much as we could without truly being a cloud.
Raspi cannot be used for a semi serious server application due to microsd card write wear issues. All other methods of attaching external storage by USB bus to an rpi are not reliable or suitable for this sort of purpose. Nowhere near the longevity of something with a native sata or m.2 SSD interface and a real SSD.
There's also the added complication of the fact it's an ARM based device so in some cases you might run into compatability issues, or testing your container locally becomes problematic.
"Raspi cannot be used for a semi serious server application due to microsd card write wear issues."
As an end user, I run a personal authoritative DNS server that has small RAM requirements. The RPi (or other SBC) boots to a mfs mounted root, then mounts all directories as tmpfs. Then I remove the SD card.1 As such, the logs for this server, which are automatically rotated and do not exceed 5M in total, are written to RAM.
1 I only use the SD card to boot. The only files on the card are a bootloader, a bootloader config and two kernels, each with an embedded filesystem. If updates are necessary, I make them to one of the kernels at a time. The other is the backup. The bootloader and bootloader config lets me specify which kernel to boot.
Don't most modern SD cards contain some wear leveling logic?
I did a bunch of research on this a while back and the conclusion I found was: yes, but buy name-brand better rated cards to avoid cheapo cards that do not do wear leveling or do it very badly.
It's nowhere near as good as SSD drives, but it's better than just a naked flash chip... unless I'm wrong.
Hey if you see this can you answer one more question:
How much abuse was your high availability product delivering to these cards in terms of writes? Was it something like a video recorder, database, cryptocurrency, or some other application that did large amounts of write I/O?
I ask because we're about to ship something that uses SD cards, but the I/O is very low. It's a network appliance and doesn't do anything locally that is high write throughput.
The Linux kernel in raspbian treats the SD card like a normal block device. I'm not aware of any special optimization for write wear leveling at the OS level, or on the board's sd card controller.
SD card wear-levelling is usually supposed to be handled on the controller inside the SD card. SD cards aren't like having raw access to flash, you basically write over SPI and the controller in the SD card decides where to actually put it.
This talk about SD cards is really great and explains some of this:
Working with RaspPi and Docker has been surprisingly difficult. Docker's support for multi-arch is poor at this point, and also applications that are I/O heavy seem to bottleneck at the network.
On a related note, Chick-fil-A using Deep Learning models for object detection with MXNet to track how long fries have been waiting: https://www.youtube.com/watch?v=3Uuq_cX8b1M
It would've been much simpler to have a plate with multiple pressure sensors and a multi-color LED per spot. When you set the fries down on the sensor it activates and the LED turns green. After X amount of time the LED can turn yellow, meaning that it's becoming stale. Finally, after X+N time has passed and it's no longer fresh it can turn red. Removing the fries turns it off. Aside from being way cheaper, I'd conjecture that this would be much more reliable. I'm pretty sure I could get a prototype of this up and running over the course of a weekend or two.
If you wanted to get really fancy I guess you could also track the room temperature and moisture levels and use that to get a better guess of how long a group of fries will remain fresh. Although I don't know if environmental factors like these have enough impact on fry freshness to be worth taking into consideration.
Anyway, it looks like they were just doing this for fun and learning, so I guess it doesn't really matter.
This was a good learning experience for our engineers at our Innovation Center at Georgia Tech in Atlanta. It may bear fruit in the form of a useful solution in the future, though. What is unique to Chick-fil-A is 'volume'. We do a lot of sales in our restaurants, so anything we can do to try and make our team members lives easier is important to us. We want them to enjoy their jobs and we want to do the best we can to consistently create high quality food experiences for our customers. Our teams in restaurants are the heroes, but we are trying to use technology to help them do what they do. <thumbs up>
The computer-vision based version means you just take the first fry off the rack. Yours means the employee needs to assess the LEDs each time (which may only take a second, but is repeated hundreds if not thousands of times a day), reach more for the yellow one in the back, etc. That's a lot more sensors to break down than a single camera, too.
Would just like to say Chick-fil-A should publish more like this. They are absolutely knocking it out of the park with tech. My wife and I are friends with a few operators (they don't have "owners"). I recall them, several years back, trying to get me to quit consulting to come work with them and open more stores. "Food service, ugghhh, no offense" I said. A friend whipped out her phone and showed me their really great in-house app that showed detailed analytics for her store (everything you could imagine) and tools for for doing purchasing and much more we didn't have time to get into. This is circa 2010 if I recall. For reference the iPhone SDK dropped in late 2007 and these folks were already way onboard with mobile. I was impressed.
I see people like this the way I see my Mormon family. They usually try to be very kind and I value that.
At the same time, they harbor beliefs which are incredibly damaging to both me and those I love, so I keep a healthy distance while understanding that they mean well.
Don’t mistake their kind acts for open-mindedness or acceptance, though.
Wait, so in order for a person to be open-minded, they have to not only honestly consider other ideas and perspectives (which is what being open-minded actually means) they also have to agree with your perspective?
A person's open-mindedness is independent of whether or not somebody else agrees with them. Otherwise, by definition nobody - including you - is open-minded, since there are people that disagree with you. It's completely nonsensical.
I'm fascinated by your stance - genuinely fascinated, I'm not trolling you, I'm serious.
So to recap, the CEO of Chick-fil-a is part of a group of people ("they") who are actually unable - as in, incapable - of honestly considering certain viewpoints? How did you reach this conclusion, and how did you come to be so certain that this CEO is in that group?
Is your understanding that he gave no serious thought to his opinion (i.e. it was just imposed on him), or that he thought about it lightly but lacks critical thinking skills, or something else?
Is there any possibility that he simply has a different opinion than you, and that's all there is to it? Like, he educated himself like you did, thought about it like you did, etc. but just reached a different conclusion than you? Or is the only explanation that he is flawed/limited in some way?
Apologies in advance if anything I've said comes across as demeaning, I'm genuinely trying to understand. Thanks!
FWIW it's virtually impossible to determine the open-mindedness of someone from afar, and to make assumptions about their ability to even consider other ideas seems like a way to insulate oneself from difficult ideas - by declaring that someone is incapable of an honest evaluation of your perspective, it prevents your perspective from being scrutinized (which, ironically, could lead to being close-minded if you're not careful).
A parting thought: most people
(a) can be reasoned with if not attacked - you can find common ground and have a good discussion with nearly anybody
(b) on every "interesting" social/political issue there are intelligent and thoughtful and just really great people with nuanced and reasonable perspectives all across the spectrum - the extremist nutjobs on the fringes are wildly outnumbered by pretty normal people
So as a random dude on the internet, just a friendly warning that if the above strikes you as untrue (e.g. if you believe large chunks of people are intolerant or incapable of being reasoned with, or if you see a difficult social issue as pretty much black and white and can't understand why so many people don't get it), then there's a good chance that you've been duped and have been sucked into a form of modern tribalism.
>My belief is so self-evident that anyone who comes into contact with it must agree with it, therefore if someone doesn't agree with me it must be because they haven't come into contact with it, and the idea is everywhere so therefore if they haven't come into contact with it they must not be open-minded!
I don't know why, but for some reason the juxtaposition of traditional, brick-and-mortar brand images with tech blogs about building modern application technologies is amusing to me. It's seeing two completely different sides of the same organization. There's 0 condescending sentiment behind it, it's just like "wow, that was unexpected."
I'm in Atlanta and have had a few discussions with people at Chik-fil-A (CFA) about joining on of their teams. I have a good friend that just moved from Intel to CFA working on business analytics. It seems like a really great place to work. And yes, whenever I mention to someone that I might be working on machine learning problems at CFA I always get a puzzled look in return. :)
I worked for Nordstrom for several years and whenever I describe their technology department to another developer they're always pretty shocked. It was pretty shocking to me when I started there too.
I was surprised too, when I saw their name in the Devops Handbook; apparently they were one of the pioneers of Devops. Talking with a previous employee, it sounds like they really reinvented themselves from an IT department to a little tech oasis within the company.
You can find some pretty awesome work on "traditional" companies. There are some pretty cutting-edge projects where I work too :) (but way more legacy migrations and boring financial CRUD apps).
I think one thing that doesn't always occur to programmers who haven't experienced is that really boring problems can be really interesting to work on.
(And really interesting problems can be really boring to work on, and really interesting companies can have a lot of cruft and technical debt under the hood.)
Their tech stack is similar to what you'd expect from any large tech company. I can't speak much to the server side of things (I worked on the iOS app) but I know they had everything ranging from older monolithic .NET apps to Node and Go micro services.
Their GitHub page (https://github.com/Nordstrom) doesn't have much in the way of internal projects, but it has lots of forks that will give you an idea of some of the tools they used.
To add to the curiosity, how is Nordstrom doing anyway, compared to other mall-type retail? I'm wondering if this forward-thinking in tech is allowing them to successfully fight "new" ecommerce such as Amazon
As far as retail companies go, they’re doing well. Stock isn’t doing great, but as far as I know they’re still quite profitable.
Being forward thinking in tech has helped them a lot with regards to being able to move pretty fast on new things. At the same time, a website as old as theirs has a LOT of legacy.
What's interesting to think about is how, in the lead up to the dotcom bubble/bust, I would think execs at tons of traditional retail/brick and mortar/consumer brands would have probably been like shit shit and then put more IT-focused organizational architecture components in place, which probably lead to many interesting technical solutions to many of the problems they faced (if they did it the right way). It would be cool to see different branching that lead to different technical solutions for common problems.
Unfortunately, as this was before the OSS boom that came later, we probably will never see these solutions out in the open.
The 4 hour workshop/hello world for AWS, it was in the intro deck on why you should be doing analysis and what not. Probably preaching to the choir for most of the room.
Most retailers don't have the challenges of scale that we have and I think thats why its weird to see tech innovation in a restaurant. That said, we need it to scale and are grateful to the k8s community for making what we are working to do possible!
I enjoyed reading this post, but came away with more questions.
They talk about providing a service for developers to deploy stuff easily, but I'm wondering how that works in practise.
I've very little experience with Kubernetes, the author mentions they run 'mini' kubernetes clusters at each restaurant. Does that mean they have to deploy software/container updates to each restaurant one by one? Or is it abstracted above that level, where they can see ALL of the restaurants as one "big" cluster?
Hey! Caleb here... I'm the lead SRE building the clusters on the restaurants.
Each individual restaurant gets it's own cluster (we aren't fully deployed yet). There are too many network latency challenges and too much immaturity around federated clusters to take the route (unfortunately).
We currently use gitops, which houses all the configs per cluster in a single repo per restaurant (CRAZY amount of corgis!)... we call that Atlas (the repos) and we made a little pod called Vessel that polls and applies configs in those repos.
We're almost done building something called Fleet that will generated and manage those repos (Atlas) at scale. Ie; a UI where we can say "send this version to 10% of restaurants" and it will regenerate and deploy those configs to all the appropriate Atlas repos, which will then get pulled down by Vessel.
We tried doing all this with Helm but failed miserably. Maybe it was us? But templating vs gitops... the choice seemed obvious.
Caleb, we’re Armory, W17. This does indeed sound like an interesting use case for Spinnaker, as Jacques said. Happy to chat with you about it. DROdio@Armory.io (we are commercializing Spinnaker — http://www.Armory.io )
The primary challenge was just reasoning with the template and using Helm at scale... ie; what exactly did we deploy on those hundreds of varying clusters?
Other issues included; tiller would sometimes become unstable... version mismatch issues between helm local and roller... lack of a clear, outage free canary deployment... we even found cases where helm would not cleanup after itself during a deployment and retain previous config settings within k8s.
For me, Helm caused more problems than it solved. Pulling in packages always seems good, but as soon as you want customization, you're back to merging in the (relatively) straight-forward yaml files from the chart. Also, instead of Helm's templates (which get crazy complex and unintuitive), a simple tool like Kustomize[0] is very straight-forward and allows per-environment configuration. Finally, tiller pods do present (yet another) security risk for Kubernetes.
We should have a Medium post about this soon, but we built a set of tools called Fleet, Atlas, and Vessel. The simple premise is that Fleet generates code for 1..n clusters. By code I just mean yaml files. Atlas is a git repo per cluster that holds the declarative state for that cluster. At any time it should be able to be applied to the cluster and result in the ideal state. And then vessel runs in each cluster and does a (I am oversimplifying a bit) git pull and kubectl apply -f. We can use these tools to do canary deployments (generate x percent this way, y percent that way in ingress rules) across a wide range of clusters. The goal here is really to minimize configuration drift and keep a large number of fairly ubiquitous (but not always so) clusters in order.
This is Brian -- I wrote the article. We are closed on Sunday but its not dogmatic in nature. We want to make sure that everyone at Chick-fil-A has a great work/life balance. Occasionally we need to do some work on a Sunday but we try to architect and engineer in a manner that supports business-hours-friendly deployments, or scheduled deployments over night to protect that. This is truly a first-class design consideration. In my 14+ years at Chick-fil-A, I have only worked a handful of Sundays for any reason. Its a tribute to leadership at all levels and a great company culture that values people (and their need for times to rest).
I happen to agree with the views they have stated in the past. Marriage (defined as being between one man and one woman) is one of the most foundational institutions in the world and is worth defending.
My marriage is just as valid as anyone else’s. And it’s incredily offensive to suggest that my marriage is somehow an attack on marriage itself. Your view is bigotry in an old, familiar form, and can only be defended based on religion, which has no role in limiting civil rights within our secular society.
Since we have a highly available Kubernetes
cluster with data replicated across all
nodes ..
This sounds like one of the more cutting-edge bare-metal kubernetes setups I've heard of!
Curious what the kubernetes implementation / configuration looks like for the setup they've described. Particularly in terms of HA-cluster k8s configuration, networking solutions integrated, and replicated storage.
Considering there is no AWS / GCE / Azure block-storage primitive to build on, it'd be cool to understand the choices and tradeoffs made in this scenario.
Is all data being stored on something like HDFS or CockroachDB?
Or perhaps the k8s minion slave nodes launch containers with Ceph volumes.
Well, it’s a 6 man team so most of our trade offs were for expedience, not the worlds greatest architecture. We cheated and sync’d data using a HA MongoDB setup amongst the cluster. RKE for K8s clustering was a life saver (nothing easier at this point on bare metal IMO), although RKE can be brittle at times.
Hi Jay -- this is Brian (I wrote the article). There are constant tradeoffs. I am leery to say we are the most sophisticated bare metal setup. I think all of us that are close to the work would love for many parts to be much more sophisticated. That said, the way we handle distributed data right now is using Mongodb. We have a cluster that spans the three nodes that we have in a restaurant. The primary use case is persistence for our MQTT topics. Most of this data is what I will call "mildly ephemeral". It isn't needed for the long term, but we do `hope` to maintain it long enough to exfiltrate it to the cloud and deliver it to any necessary recipients. If we were to have a global cluster outage, it is not the end of the world. Over time, it probably will be. Back to work! :)
I've never been to a retail establishment with anything close to the credit card speed of CFA. The cashier can swipe and grab receipt in one smooth and quick motion. It's so fast you feel like it's broken or slight of hand.
I wonder how they do it or why other companies can't or don't do it. CFA stores are so much busier than even an average McDonald's, maybe the efficiency gains are just worth more.
From what I understand, merchants can do a quicker swipe (basically record the card # and transaction amount without waiting for authorization), but the merchant fees are slightly higher because of the possible higher fraud/chargeback rate. It's up to the chain to decide on what to do.
Perhaps CFA can do it even better since they could possibly keep a database of bad cards at the edge. Remember those books that cashiers used to thumb through to verify if a card was stolen?
If it's a repeat customer, at the same location, then you can avoid contacting the payment processor gateway, and just record the transaction on the local cluster for later payment processing. The expected value of the loss if someone stole their card, and used it at the same restaurant that they frequent as a regular, after the card was canceled, and the restaurant gave the thief a chicken sandwich, and now corporate needs to eat that cost because the asynchronous CC transaction doesn't go through when they finally run it... is absolutely miniscule.
The one place where CFA IT seems to fall down is at the drive-through.
I used to be able to pay for my order with the order taker on his/her iPad using my Apple Watch.
For the last six months or so, I have to go all the way to the pick-up window, and they have to get a manager to unplug a card swipe terminal from the front counter, plug it in near the window, log in to the terminal, and then hang the actual terminal out of the window so I can tap my watch.
Very strange. Hopefully it's only a problem at the one CFA in my geographic area.
Sounds like it would be fun to be on your team, but my work lies on the other end of the equipment chain. I do OEM work for the fryers and grills, and boy could they use a kick in the pants from the chains to get their act together.
Hey this is Brian -- I wrote the article. I have heard the term a lot over the years. I think you could call it that -- the idea being its the "foggy" area between the user and the cloud. The reason we like "micro private cloud" is that 1) its a tiny environment with limited cloud properties (no elastic scale) but 2) we do try and provide a platform with cloud-like services for our internal developers, and we like the cloud paradigms being applied to how we think about compute at the edge, in restaurants.
Because more than one person on the team has a background in meteorology, and to them a localized cloud event is "fog." I think it started as a joke, but it just kind of stuck.
It helps that our events that require localized cloud infrastructure are sporadic, just the way real fog is. And just like real fog, we don't always know where it will be. Once or twice a year we have to deploy about 150 devices to a different location in the desert. Then as quickly as we arrived, we pack up and move on.
This is a pretty cool example of cybernetics ala Stafford Beer - constant information and reaction to streamline production. I'm excited to see this be brought in to more production (though maybe it has, and most places aren't as upfront as this). It's truly a shame the Chilean experiment in the early 70s didn't work out.
On a side note, it's interesting that Kubernetes is the word from which cybernetics is derived, meaning "governance".
Hey this is Brian -- I wrote the article. It really is an interesting time, especially when you have the scale that Chick-fil-A has. While traditionally a fairly archaic industry (POS registers, ERP at corp, thats about it), the volumes and challenges of scale have created many challenges that solid engineering can help solve.
the domain is interesting, but I worry about the long-term effects of maximizing efficiencies (read: automation) forpeople who rely on jobs in that industry.
The whole debate on if automation is a good thing or not has always been pointless to me.
It's like trying to un-discover nuclear weapons, once it's out there you can't go back. Things are going to get more efficient, more automated, and those jobs will be lost. It's not a matter of "if", but "when".
What we need to do is to figure out ways to cope with the change. Find tasks that these workers can do that allows them to earn a living, experiment with universal income to maybe one day not need everyone to work, fund education programs so these people can learn to do something different, provide some safety nets so that these people have time to cope with the changes.
We need to adapt as society, not pretend that this technology doesn't exist.
> What we need to do is to figure out ways to cope with the change. Find tasks that these workers can do that allows them to earn a living, experiment with universal income to maybe one day not need everyone to work, fund education programs so these people can learn to do something different, provide some safety nets so that these people have time to cope with the changes
A myth about manufacturing jobs is that they all went overseas. Reality is that most of them were replaced by robots and automation.
Low skill, low knowledge, repetitive work is key target for automation.
Fast food workers wouldn't have large downstream effects. Automated truck driving would though. There are likely entire towns right off of highways that depend on humans passing through daily - restaurants, repair shops, etc. If automated trucks don't stop anymore, a town could literally be a ghost town.
Hey this is Brian -- I wrote the article. I do want to add that Chick-fil-A isnt looking to replace jobs with "robots". We really want to assign our restaurant team members to tasks that add maximum value to customers, which are generally service oriented (iPad in Drive Thru, personal service in the dining room, etc). Our automation will help our team members be more effecient and enjoy their jobs more rather than replace them.
Of all places I doubt CFA would replace workers, having nice, helpful people is the second awesome-est thing there (chicken and waffle fries tops that).
I can see the automation and analytics helping staff appropriately and offer more options for efficiency and waste reduction.
I think you hit the bullseye here, because the tasks you highlighted simply aren't done at most other QSRs. They are novel ways to apply human labor to fast food (at least, novel within their competitive space).
I may be alone on this one, but if Chick Fil A automated most of their workers, I would stop eating there. Part of the reason I go there is I am always impressed at just how good they are at customer service. There isn't another fast food restaurant that even comes close that I know of.
I hate talking to machines, I hate talking to automated voice menus instead of getting actual assistance from a human being. I hate ordering stuff online and dealing with slight one off cases. The google Ai assistant had me cringing for weeks. I especially have a deep hatred for self checkout lines. They can all burn in hell.
Why? I think that they move on to another service industry job such as customer service or something. I don't think working in a fast food restaurant is building the sort of skills that making a horizontal jump to another industry is outlandish.
I think that they move on to another service industry job such as customer service or something.
That's just something people think so that they don't have to feel empathy for people who lose their jobs to automation. It's kind of an internal excuse.
The reality is that not everyone is re-trainable. And if someone could easily get a job in customer service, or some other marginally better situation, they already wouldn't be working at a fast food joint.
In addition, many fast-food jobs are temporary/seasonal. A high school kid can get a six-month stint at MacDowell's to pay for car insurance over the summer. A call center isn't going to take him in and train him knowing that he's going to bolt come September.
Recently, I was contemplating something regarding customer service jobs and automation. I was at a fintech data conference listening to the work being put in to analyze customer support call log history to understand where prospects are in a pipeline or improve retention.
If a system can pinpoint a prospect is close to onboarding, and customer service rep is a great closer, the software could pass the prospect off to the "closer". The same could be said for keeping a customer who is on the fence from leaving. All of the sudden, a highly skilled closer has data and metrics attributable to the company's bottom line, and in theory, someone like this should become a high priced asset.
In some instances, could automation actually greatly increase the salary dispersion in certain low wage fields?
> I think that they move on to another service industry job such as customer service or something
Right, they can compete with all the other displaced workers for the #1 position most likely to be outsourced.
Hell, many Targets don't even have a customer service desk anymore. They demolish them and assign the workload of both sales and returns to a random cashier.
A lot of people are not qualified for skilled work and will never transcend entry-level fast-food jobs. If you take away those tasks that keep them productively occupied, it might be a good time to start buying stock in industries related to prison management.
There are certainly negative consequences of automation, and many existing jobs will disappear.
A key feature of the times we live in is that knowledge is ubiquitous. If an idea becomes known and understood to be feasible, and it serves a useful purpose, you can't hide it from existence. If your restaurant doesn't capitalize on these tools to improve efficiency, another one likely will. This might mean that your restaurant gets be out-competed by a better product elsewhere AND your workers will be out of a job.
Trying to freeze a system that works at a given time is becoming more and more difficult as we advance our knowledge and technology. This paradigm of rapid evolution has been playing out since the industrial revolution, and seems to be accelerating with time.
Maybe it ends badly for us in the macro sense, we'll see. I feel for the folks that can't evolve and get left behind, but I'm not sure how to help them. Resisting the change probably won't work.
We would probably be okay, if it wasn't because we are also looking at soonish fully automatic trucks, which will kill all the jobs in the trucking industry, plus all the jobs at the trucking stations, mechanics etc, plus the retail collapse.
I guess if you aren't in one of those areas, and you have a business that hires a lot of low income people, times will be good, but like you I don't think the rest of the market can take up the slack; there are only so many people who are willing to pay for laundering.
From other retail IT I've been involved with, these guys could make an incredibly successful side business doing this to pretty much any retailer.
However when I see that retail stores are using dialup to do batch uploads of inventory and transactions I wonder if their central IT systems would be remotely ready for something like this.
Interesting that they use Kubernetes exclusively for edge computing resources (as opposed to their cloud architecture). Never really considered this approach before.
We use K8s in the cloud as well! Our entire control plane for the IOT activity runs there. The rest of the engineering teams have a mix of solutions, but many are moving to K8s as well.
Your comment is essentially "the owners hold certain values I disagree with regarding sexuality, therefore I'm surprised their technical team is any good."
Political values are important to point out. Companies create these tech blogs as a recruitment vehicle. You should know if the company you're considering applying to is funding bigotry (or "wholesome family values" if that's your angle).
Of course neither of those things is actually bigotry. This "oh I'm suffering" crap is disgusting. If we actually cared about suffering our freely elected government wouldn't be causing so much of it.
Euphemisms like "advocating for his personal beliefs" wouldn't fly, if say, the owner was donating to organizations that fought to make interracial marriage illegal again.
I love seeing this argument. It's like you people think you can raise any arbitrary instance of hate/intolerance/bigotry onto a criticism-proof pedestal simply by brandishing a dictionary. Like, who are the real bigots: Nazis, or the nasty people who have the gall to disagree with their racist, antisemitic, etc. views? Is the answer not obvious? Does the question send you scuttling back under your dictionary?
I'm fine with a bit of pedantry but ultimately in the real world we need to show a little more intellectual maturity.
You're missing the point. I'm not concerned about the word. I'm concerned about the definition - those quickest to cry bigot are often guilty of acting like one themselves.
Case-in-point: Twitter CEO Jack Dorsey being forced to apologise for eating at Chick-fil-A during LGBT Month. What kind of bizarre intolerant nonsense is that?
Another case-in-point: Boston and Chicago mayors claim "Chick-fil-A values are not [our] values" and refuse to let them open branches in those cities.
> I'm not concerned about the word. I'm concerned about the definition
Come again?
It seems like you're the one who's missing the point, anyway. You didn't address anything I wrote in my comment. Par for the course, I guess. Just imagine I wrote the exact same comment again, and reply to that if you feel like it.
The paradox of intolerance[1]: being tolerant of those who are intolerant leads to less tolerance. It is completely right to call out intolerant views and actions and work towards eliminating them.
The obvious problems with 'the paradox of intolerance' being that anyone who can convince themselves of another's intolerance now has moral license to behave intolerantly to an unspecified degree. Of course, exactly this sort of thing happens all the time; notably, it was used to rationalize Google's firing of James Damore--Damore's memo allegedly contained "dogwhistles" so he deserved being fired, slandered by the media, and mobbed on social media.
If it wasn't problematic it probably wouldn't be called a paradox.
I'm inclined to divide intolerances into two types: intolerances based on what somebody thinks, and intolerance based on what somebody is. I'm further inclined to model intolerances as a directed graph where nodes represent a group of one of those two types, e.g. conservatives and men might be represented by individual nodes of those two types respectively. Edges represent an intolerance, e.g. there would be an edge pointing from Klan members to black people. The former node would be of the first type, the latter of the second.
Where there's a cycle in this graph we are seeing a mutual intolerance of people with different ideas. An example might be people who think the Earth is flat and people who think it is round. Who is right? It's impossible to say without diving into the issues themselves.
flat earthers <=> round earthers
Because what you think is independent of what you are, nodes of the second type cannot have outgoing edges. Incoming edges represent base intolerances of the sort that have historically been very harmful: racism, sexism, homophobia, and so on. Obviously these edges represent a different type of intolerance because they are based on innate attributes that their targets have no control over. To a liberal person, they are essentially definitionally bad. To an illiberal person, they're fine. Great, even, depending on who you ask.
Note that I'm talking about liberalism in the general definitional sense, not in the modern U.S. politics sense (i.e. "the left").
So, how about this graph taken from real life (the <=> represents two edges, one pointing in each direction):
anti-racists <=> white racists -> non-whites
Deletion of the rightmost edge (i.e. deletion of white racism) means that we also have to delete 'white racists' or the implied meaning of the graph is paradoxical. It then follows that we have to remove 'anti-racists'. Now we have no intolerance at all—all that's left in the graph is `non-whites`, enjoying their existence without harassment, and all it took was the elimination of racism. You could also remove the 'white racists' node, with similar meaning and identical effect.
However, deletion of 'anti-racists' leaves the racism intact. I'd argue that this form of graph appears in many places where there is an illiberal group intolerant of a type two group and a liberal group locked in mutual intolerance with the illiberal group. The inverse setup doesn't really appear because, as I noted before, edges pointing to type two nodes represent a sort of intolerance that is anathema to liberals.
I'd also argue that there is a strong implication that these sorts of edges are the most important to remove—because, as demonstrated, doing so causes the rest of the graph to collapse. They are the "root cause", if you will, of many webs of interconnected intolerances.
It should be obvious that this model denies the assertion made by many modern American 'conservatives' that intolerances are all equivalent and relative, "the tolerant left", etcetera. I would say the water is most muddied by the fact that our 'left' and 'right' do not map cleanly onto liberal and illiberal. It's easy and sexy—but invalid—to apply the same political label to someone who thinks gay couples should be able to enjoy the same legal benefits as heterosexual married couples, and someone who thinks all white people should be killed.
P.S. Damore is a moron and his pseudoscientific screed was legitimately offensive to his co-workers. That's fine justification for him to be fired from his job at a private company. As for the rest of it, we live in a culture of growing illiberalism and zeal for generating outrage. It's a bummer.
I think you missed the point of my post, which was that this paradox allows anyone to abuse anyone else if the abuser can convince themselves that the abused is really 'intolerant'. As in the Damore case, a certain political group didn't like his message, so they claimed it contained hidden white/male supremacy messages, which was sufficient to justify his termination and abuse (contrary to your ad hominem post-script); the mob patted themselves on the back citing this very paradox.
I think you missed the point of mine. Doesn't matter.
A lot of people made a lot of politicized fuss about the Damore thing, but the simple fact is that his document contained offensive generalizations about women and if I was a woman I would not want to work with him. He's free to say whatever he wants; as I said before, though, this is a fine justification for termination.
A lot of people from all sides of the spectrum made it a political issue, because of course they did. The left said Google didn't act quickly or harshly enough and the right acted like Damore's right to speak freely was being taken from him. I'm not one of those people and I'm not interested in discussing the issue in terms of 'ideologies'. Ultimately, as I've said before, what he did does not have to be viewed through a political lens for it to be a legitimate justification for termination.
edit: I said "if I was a woman I would not want to work with him." Actually I would not want to work with him regardless.
I may have. It sounded like you were highlighting the paradox itself, while I was intending to highlight the way the paradox lends itself to abuse. Perhaps ironically, your comments about Damore and the memo are exemplary--the memo contains zero offensive generalizations; it's been picked over and no one has produced an 'offensive generalization'--only phrases that allegedly contain offensive hidden messages. And because of those hidden messages, the mob pats itself on the back for helping to suppress Damore's "intolerance" (by the way, the same mob was rationalizing the New York Times hiring Jeong despite her four-year montage of overtly racist tweets). Again, I don't want to focus too much on any one example, because there are so many others and I don't want to steal attention from the broader phenomenon.
> I may have. It sounded like you were highlighting the paradox itself, while I was intending to highlight the way the paradox lends itself to abuse.
My intent was to illustrate that not all intolerances are only relative degrees of the same thing (as claimed in the popular 'conservative' talking point); some are categorically distinct, and in a web of interconnected intolerances the two types that I identified interact in asymmetrical ways.
I mostly just wanted to write down an idea that occurred to me on my walk home from work. I don't expect something that long to play well in an internet forum. It's meant as an interesting abstraction, not an airtight position.
> Perhaps ironically, your comments about Damore and the memo are exemplary--the memo contains zero offensive generalizations; it's been picked over and no one has produced an 'offensive generalization'--only phrases that allegedly contain offensive hidden messages. And because of those hidden messages, the mob pats itself on the back for helping to suppress Damore's "intolerance" (by the way, the same mob was rationalizing the New York Times hiring Jeong despite her four-year montage of overtly racist tweets). Again, I don't want to focus too much on any one example, because there are so many others and I don't want to steal attention from the broader phenomenon.
It's quite literally the only piece of coverage I've seen of the Damore saga that I mostly agree with. Otherwise I think I've said what I have to say about him and his document. If you want to project politics onto me, I can't stop you, but I don't think it's productive. My opinions on this issue aren't political.
> My intent was to illustrate that not all intolerances are only relative degrees of the same thing
Ah, I see. I agree, though I wasn't espousing the "popular conservative talking point" (which I've never actually heard before), so I'm not quite sure why the conversation took that particular turn.
> Have you read this
I have, but I don't really want to dissect that here and now.
> If you want to project politics onto me
I don't want to, and I don't think I did. :) Sincere apologies if I offended.
Your very own link lists other major philosophers and thinkers (including Thomas Jefferson) who disagree with Karl Popper's conclusion on paradox of intolerance! There is _no_ universally agreed upon answer to the paradox of intolerance; I do not know why people keep pretending that there is.
(I will also observe that, where the article notes: "Popper asserted that to allow freedom of speech to those who would use it to eliminate the very principle upon which they rely is paradoxical.", it clearly applies to those who wish to (quoting from your post) "...call out intolerant views and actions and work towards eliminating them".)
The quoted definition of bigotry as "intolerance toward those who hold different opinions from oneself" is from the New Oxford American Dictionary, from Oxford University Press (the OED people). If that's what Google is using, I don't think I can rely on your definition of "crappy."
Yep, it's a crappy definition that does not reflect current usage. It doesn't even reflect the definition in the OED, if you want to check there. 'Bigotry' is not limited to difference of opinion. Nobody uses it like that, you can take a look at the quoted usage examples here, which are from current publications.
Ahh, yes. Always useful to continuously cherry-pick or alter definitions to tailor words that have an objectively negative moral implication to apply to exactly the standards you want.
Just because some publications use the word in that way does not mean the definition of the word should be shifted.
I’m sorry but this is a pointedly stupid comment. I’m doing the exact opposite of cherry-picking - I’m saying the definition is much broader than the inaccurately narrow one posted.
The OED is a historical dictionary; the Oxford Dictionary of English and the New Oxford American Dictionary are explicitly meant to reflect more current usage. At any rate, I think this is getting awfully pedantic; the definition you linked is "obstinate or intolerant devotion to one's own opinions and prejudices," which seems similar enough that I'm not really sure what the point of this argument is. Given that I'm fairly sure the point of this was whether Chick-Fil-A's executives and in-the-company's-name giving exhibits bigotry, by any dictionary anyone's linked to, the answer is "yes."
The point is that the definition posted is narrow and bad. This isn't some obscure technical point, a debate about the proper usage of 'less' and 'fewer' or whether 'literally' literally means what it says in some odd case. All you have to do to convince yourself of its badness is read a newspaper or magazine or listen to a news report. What exactly would 'racial bigotry' mean if the posted thing was the only definition of 'bigotry'?
As to pedantry, you're the one who started arguing this clearly lousy definition is not lousy, brought up the OED, decided the OED is no good, finally acknowledged the existence of a broader definition but then somehow decided that it's, in fact, as good as identical to the narrow bad definition. I'll be the first to acknowledge I have a habit of getting involved in some deeply silly threads but I'm left utterly flummoxed by this Cirque-du Soleil-level display of mental gymnastics.
I just hope the authors of that lousy definition aren't holding your cat hostage.
I have a pretty personal beef with the idea that language can simply be contorted to supposedly mean whatever someone feels like it should mean, as heading down that road is basically abandoning meaning and communication altogether, and is wholly incoherent.
For those coming to the thread later: the parent comment pointed out a certain fact[1] about the corporate activities of Chick-Fil-A that doesn't directly relate to the contents of the article.
The basis was the observation that such articles are,in essence, a "work-for-us" ads, and in that vein, informing the readers of other, broader actions of a corporation is relevant, especially when it pertains to ethics.
To make the point more directly for you, I think and I’m guessing the poster thinks we ought not to normalize certain kinds of discriminatory corporate behavior by just looking away and cheerfully admiring their oh-so-efficient tech stack.
This whole post and thread are an effort to recruit engineers and accumulate good will within the HN community. Pointing out that the company is rotten is extremely important, as is anything else that can undermine the project of manipulating talented, powerful hackers into working on the project of generating profits that power an evil social agenda.
Hey, this is Brian and I wrote the article. I have worked at Chick-fil-A for over 17 years. I will volunteer time to take a phone call with anyone that is interested in hearing my first-hand experience with Chick-fil-A and its culture over the years. Just reach out on LinkedIn if you are interested and we will schedule some time.
I'm sorry, but whatever positive experience you've had as a member of the organization does not simply negate the negative effect the ownership has on our society.
Yeah, me too, but at least I wasn't writing Medium articles about how cool it was that I was generating revenue to be funneled directly into conversion therapy youth camps.
We also have an open dialog with Kim Jong-un; doesn't mean we should applaud the people that work and support him or feel sorry for those workers. They're traitors to the human race, in just the same way that Chick-Fil-A workers are directly responsible for what the company does.
No I'm equating the CEO Dan T. Cathy to Kim Jong-un. This guy's a scumbag who loves nothing better than commanding the world to follow his hypocritical commands, and his staff are personally responsible for supporting this behavior by going along with it. I can't say I have any respect for him, his company or the people who work for him. The US should simply outlaw his antisocial policies because that's the right thing to do.
If you're going to complain about the ethics related to Chik-Fil-A, it would probably be better to start with the fact that they raise and kill a billion chickens every year rather than that they have donated $120,000 to organizations like the Salvation Army (example from above article).
Chik-Fil-A is not an arbiter of LGBTQ rights. Their small 6-figure donations to organizations that focus on other issues do very little in the grand scheme of things.
Do you really think Chik-Fil-A is directly causing more suffering by donating to a few charities in amounts this small than by raising in captivity and then killing billions of chickens? I see no way to reasonably argue that.
Chick-Fil-A is just the service provider, if not for Chick-Fil-A people would get their chicken elsewhere. If they were raised in a place that felt less like captivity, that would probably be worse for the environment, in terms of using more natural resources per pound of chicken produced. There are trade-offs to everything.
Chick-Fil-A isn't really in the meat business, they're in the restaurant business, and given that they aren't in a very tiny niche, their menus feature meat. They sell plenty of soft drinks and fried potatoes which reduce the amount of meat as a percent of the total calories.
Finally if you're going to get people excited about animal rights, it's probably better to mention cows than chickens. To me, Chick-Fil-A's ads suggest eating chicken as a lesser of two evils, without ever saying that eating chicken is evil. We also don't have a separate word for chicken meat like we do for cow or pig meat, which I think serve as euphemisms.
You may well already know this, but the separate words for meat and animal are likely because of the Normans; the richer French speakers cooked and ate the animals, so they got French names; the poorer "English" raised the animals and called them by their English/Germanic names.
We do have a word with a French root for chicken, by the way: poultry. As to why the separate name didn't catch on, I can only speculate that it's because chicken are cheap to keep and so the English may have eaten them themselves.
Not all languages have separate words for meat - German perhaps being the most famous. It's slightly disconcerting to walk around a supermarket and see schweinefleisch being advertised.
> Finally if you're going to get people excited about animal rights, it's probably better to mention cows than chickens. To me, Chick-Fil-A's ads suggest eating chicken as a lesser of two evils, without ever saying that eating chicken is evil.
I posted this article to our company #random channel and that was the first reply- better chicken than beef!
Most effective altruists would agree with you (and thus disagree with a possible takeaway from Chik-Fil-A's advertising) that eating chicken is better than cows, as cows provide significantly more meat than a chicken, with ratios being up to 50 chickens per every 1 cow, as far as quantity of meat for normal portions go. In other words, someone eating only chicken may kill X chickens, but someone eating only cows may kill 1/50X cows. Of course there are other variables involved like weighing the differences between these animals, their treatments, lifespans, etc., but it is nonetheless a point with notable weight. Although this isn't quite the area I was getting at, just that the most significant things should be looked at first, all else constant.
pretty sure the ads are just a joke to not eat hamburgers, but to eat chicken whether it be nuggets or sandwiches or whatever which is essentially the menu for Chick-fil-A.
If I thought Chick-Fil-A shouldn't be supported for either their treatment of animals or for their anti-LGBTQ corporate giving, and wanted other people to stop supporting the business, I would focus on the latter in my messaging, because I suspect it would be more effective on the audience. Attempting to draw conclusions about which of their behaviors I personally think cause more suffering from this is not a good use of your time.
If your audience cares significantly about the charitable donations and not at all for animal suffering, and your goal is to only to convince them to be against the company, then sure, go for it.
But any effective altruist should have their goals set higher than this, at actually reducing the insane amount of suffering that's created. It does very little good if everyone stops supporting Chik-Fil-A only to flock to another chicken company, which will be the same in almost every way, possibly minus some donations, possibly one which treats animals even worse. Effective altruism is a very different beast than simply appealing to or signaling the crowds' most popular views on ethics in order to persuade.
Chik-fil-a is only a useful example in so far as how well the ethical throughlines match things we've seen many times before, the media has taught us to care about, and how much time and effort political interest groups have spent highlighting those to the public. Ethical thinking on this topic will not be engaged, people will take the opinion their political affiliation and chosen memes have predestined them to take, and they'll act like it's serious thinking.
Parent poster said "Imagine being so dead inside you feel perfectly fine, even proud of working for Chic-fil-A, lol." Substitute in Facebook, Google, Apple, Microsoft or Twitter, and you might be approaching an interesting statement.
Where the ethics gets engaging are the day-to-day business model at any currently big or hot software company. Only a small subset of people have the background to comprehend the issues. And the people saying "let's stop and think about the consequences of this" gets steamrolled by the status quo that has money to make. That's where all the good examples live.
But it's a bit harder to come to terms with when the focus hits close to home. So we have whipping boys like Chik-fil-a. "At least I don't work THERE"
I agree that there are many important ethical questions that are bypassed in the day-to-day of those companies for sure, but at the same time I think there is an attitude here of "who cares what happens with the company's money once it leaves my area, they are solving interesting problems and the people working for them are doing intellectually engaging work".
No, the grandparent's political stance is to be outraged at anybody he/she disagrees with and has an unhealthy obsession to blurt it out on discussions that aren't political.
All discussions are political. The only question is whether or not the politics at hand affect you.
And to quote Chick-fil-a's CEO: "Jesus had a lot of things to say about people who work and live in the business community … Our work should be an act of worship. Our work should be our mission field." So he sees everything they do as political as well.
HN won't let me reply to your next comment, but regarding:
> a reflection of their religious principles or their political ones
This is actually a common misunderstanding and what I'm trying to explain. "Religious principles OR political principles" means there are actually (at least) TWO categories of ideas a person holds, with specific items (homosexuality, death penalty, etc.) falling inside one of those categories.
That's a secular worldview, and it isn't how Christians think. We have one category: God. God is sovereign, he owns everything including me. As Thomas Aquinas said "all truth is God's truth". When we're being consistent (key caveat), we approach all issues as "what does God want me to do?". Political activities follow from there. We often disagree about those secondary conclusions (what God wants from us regarding issue X), but not the core starting point (ultimate allegiance is to God alone).
> This is actually a common misunderstanding and what I'm trying to explain. "Religious principles OR political principles" means there are actually (at least) TWO categories of ideas a person holds, with specific items (homosexuality, death penalty, etc.) falling inside one of those categories.
> That's a secular worldview, and it isn't how Christians think
> So he sees everything they do as political as well.
Obviously we have political views, but they're a only subset of religious views. By analogy, geometry exists and is a legitimate field of study. But if you said "all mathematicians think about is geometry" that would be false because it's too narrow in scope. If he's consistent in his Christian worldview "he sees everything they do as religious as well" would have been true.
Was funding a lobbying group to reinstitute the death penalty for homosexuality in Uganda a reflection of their religious principles or their political ones?
One thing I question is what kind of WAN link a single chicken fast food restaurant can get, or is economical. A lot are in places where the only last mile ISP options will be adsl2+ over copper or docsis cablemodem. I don't see it being a good use of money to spend $600/mo to pay some local ISP for a dedicated 100Mbps line on a 5 year contract.
But then again it's highly likely that what they do is not bandwidth intensive and will work fine on a 5Mbps x 1.25 Mbps DSL connection.
One thing I question is what kind of WAN link a single chicken fast food restaurant can get, or is economical
At this point, I don't think it's a problem for CFA.
CFA doesn't try to go into every little town in the nation like McDonald's. It's expansion is very deliberate and only seems to go into very high value locations, where internet presumably isn't a problem.
Moreover, landlords fall all over themselves to land a Chick-fil-a. Almost as much as they do to get an Apple Store. A landlord will make business-grade internet happen, if it means getting a CFA.
Meh. Unless you personally own a franchise, or work for corporate, this is pure conjecture.
The article did mention if a failed node is detected, a next-day restore 'disk' is delivered to the store, so obviously that is faster/more reliable than random ISP.
Not conjecture at all. Two jobs ago I worked in a real estate adjacent field, and there were repeatedly articles in the trades about this. CFA and Apple are two at the top of real estate developer “want” lists, along with Urban Outfitters, Starbucks and a few others.
It’s why when architecture firms render new commercial buildings the fake logos often look like those brands. It looks much better in financing presentations and when you’re trying to attract other tenants.
Some agents even Photoshop Apple and CFA logos onto buildings in their private (and sometimes public) presentations to make the projects look more attractive.
Just because you’ve never worked in this space doesn’t make it conjecture.
Im not 100% sure its all of them (malls and airports come to mind) but yes its a vast majority.
True I didn't consider the cable investment either. Normally the landlord pays for this but I imagine chickfila probably pays for part of the bill. That I dont know though
For an ISP, if it requires a costly outside plant build to reach a premises for a business circuit, it will usually be rolled into some combination of an upfront connection fee (NRC) and monthly rate (MRC) to recoup the costs and have some positive ROI over a 3 to 5 year term. If it's something like a multi tenant office building they may gamble and take a calculated risk to lose money on the first customer over 5 years (example: $800 NRC and $500 MRC that comes nowhere near xoveinft the cost of the build), with a new expensive fiber build, and hope the sales team can sign up more tenants ASAP.
When a McDonald’s is located on a pad in a Target parking lot, the McDonald’s franchisee doesn’t own that land. It belongs to the owner of the strip mall.
Ditto for the tens of thousands of McDonald’s restaurants located in office buildings, malls, airports, and inside other stores like Wal-marts.
It’s not the 70’s anymore. The standalone McDonald’s is far from a majority of the outlets these days.
Chick-fil-A also has a mobile app that allows you to order your food and pick it up either at the counter or in the drive through, so they need the internet connection for that.
It's difficult to estimate what value the described systems add to a location, but in the grand scheme of a location's expenses, even $600/mo is a drop in the bucket. Align those contracts with the lease on the building itself - no more problems!
We had one just open in my town and for the first 3 days, their slowest take was $6k/hour. Regardless of their margins, $600/month would be lost in the noise.
This was posted in a devops-related slack not too long ago and caused quite a stir over CFA's lack of, uh, diversity. I know it's technically off-topic and frowned upon here, but I believe the fact that they make their beliefs such a big part of the company that it deserves to be pointed out.
That level of infrastructure investment is really playing the long game, especially for a brick and mortar business. I often feel like "traditional businesses" see technology's disruption effects but not the positives they can bring. Other businesses could benefit from this example.
Not that surprised, a fully automated restaurant is every restaurant chain's dream, reduces labour costs and logistical accuracies go up at real-time speed.
I've worked in a fast food restaurant and one of the key pieces of information we used was something called a "drop chart".
This was basically a printed out excel sheet with predicted amounts of food to keep ready to serve.
i.e. between 5pm-6pm have 12 chicken breasts already cooked. People follow these charts to know how much stock to pre-cook before the hour basically.
These predictions are usually based on previous years sales for that same day/time + growth + whatever other inputs to the sheet.
It sounds like what they're trying to do is run ML models and make really fast predictions in real time to react to surges of customers. To me this sounds a lot smarter than the approach I worked with.
From my experience when the numbers were off you either generate a lot of waste product or you make customers wait really long.
I imagine the inputs to the ML model are probably something like real time sales data + whatever else.
I think you'd want edge compute here not just for latency but for availability.
You'd want to both run these models fast and make sure even if the crappy internet connection of the restaurant dies you can keep providing these numbers.
The driving need for edge computing is low-latency. There just isn't the need for it in a restaurant.
I do hope they're not doing real-time control of kitchen equipment via something in a container, containers in Linux are nowhere near deterministic enough.
This strikes me as someone who is familiar with only web technologies but unfamiliar with control systems attempting to create a control system.... completely inappropriate tools for the job.
Could you define what is not deterministic about containerized computing?
Examples of things we are controlling are timers for food, cameras that can recognize and track food items, drying machines that will automatically trigger and fry fries.
Linux is general is not a real-time system, in that you cannot get deterministic response time from it.
For example I'm guessing it doesn't matter if one of your timers are a second late due to Linux deciding to swap a process out or prioritise some upgrade check but imagine if your time constraints were a lot tighter.
Similarly if you're attempting to track food on a conveyor belt and Linux decides to prioritise the file indexer on your filesystem... oops, a lettuce has been thrown on the floor.
Our tolerance requirements for synchronicity are broad enough that we can tolerate blips like this... at the end of the day we are automating away some simple human interactions (ie; fry the fries or track the food), we aren’t performing surgery with these systems.
So far we’ve been satisfied with Linux (Ubuntu 18.04 to be exact) and it’s overall capabilities.
Also, don’t forget that cost at scale is a big factor... doing things perfectly but expensively is not profitable at 2k locations.
The point about real-time systems is not the your tolerance of 'blips' its about analysis of your worst-case timing requirements. This is what real-time control systems are all about.
Sure, you may be able to tolerate a 1 second 'blip'... how about ten seconds? a minute? do you know for sure how long Linux may put your process on hold for? Linux provides no guarantees at all.
It seems to me that what you are really creating is a traditional control system with web technologies and using the hype-of-the-day (i.e. edge computing) to justify it... with the corresponding increase in overhead and cycles.
This does not seem like a good solution to me, the only novelty here is the inappropriate use of technologies. i.e. you've managed to dig a tunnel with a spoon.
Seems like a very appropriate use of technology to me?
If they can get "good enough" results using Linux and running with technologies that the average developer is more likely to know, is it worth the investment in running a RTOS? Especially given running a RTOS will make any higher level app integration more difficult?
The benefit of running with Linux outweighs the risk of "lettuce on the floor", imo. Yes, there's probably overhead in terms of clock cycles, but there's also probably dramatically less overhead in terms of developer resources.. and when you're running off commodity hardware you can guess which overhead will cost more!
K8s has nothing to say about schedulability or real-time performance, its just not what it does.
K8s is simply a way to coordinate containers, nothing to do with scheduling processes on this level.
They could be using VeloCloud and many of those internet connectivity problems would go away. Use multiple internet links with cellular along with dynamic multipath packet optimization
"Edge" computing seems like a term invented by people who grew up with ubiquitous cloud computing. Go back in time 10 years or so, and all this automation would have been onsite by default.
You cannot seriously tell me that creating a predictive model for cooking french fries requires a lot of computing power.
They said in the article without getting super-specific that they have low-latency / high-throughput / time sensitive data processing needs in each restaurant. It sounds like they have a whole bunch of sensors tracking their inventory and food preparation pipeline. Other comments here link to other blog posts by them with more details about french fry done-ness deep learning models, for example (wacky but cool).
If you had to go across the internet to the cloud at such a high rate from each restaurant, this would probably come with pretty high failure rates and levels of flakiness. It makes sense to me to process the local data firehose in-house and keep the rest of the common high-level stuff in the cloud.
I suppose I'm struggling to see how much of a requirement for low-latency data processing exists in a restaurant.
There just isn't that much data there unless they're attempting to do real-time control of equipment (in which case, just use a regular control system).
I can imagine a requirement for logging potentially, but thats not a low-latency requirement and the use-case for edge-computing is exactly that, low-latency.
Still not seeing anything here but a dramatic overuse of technology I'm afraid.
I think you are right that this blog post doesn't lay out a specific use case to justify this setup in a restaurant- that does not appear to be the goal of the post. An exploration of problems and requirements is probably what you're looking for.
They did drop a few hints, though. They are just hints, and going by what's in the blog post, a bit of imagination is required to fill in some of the gaps. However, if you do that, I can envision a few possible needs for low-latency edge computing that are interesting and forward-looking:
- Kitchen automation. Sensors that monitor food being cooked, and help implement a pipeline to ensure quality is consistent. (example here is that neural net monitoring the staleness of fries).
- Inventory automation. Monitor and automatically restock items. Track purchases and ingredient usage in real time.
- Data analytics. Collect detailed time-series data about food quality, facilities maintenance, foot traffic, noise levels, security, etc.
These aren't standard restaurant problems. However, it's an interesting approach to scaling data analysis and automation in the physical world. All of these use cases assume the need for "real-time" data, and they assume that such a thing has business value (which is where I think your critique is coming from).
I'm not affiliated with Chick Fil-A, so I don't know how correct any of my speculation is, but that's my takeaway from reading the article.
Credit card transactions, mobile orders, timer synchronization, order receiving (tablets etc), iot devices (cameras, cooking devices) and other things planned for the future.
- Credit-card transactions don't have low-latency requirements, its 'nice' if they're quick, same as everything else. The bottleneck here is entirely dependent on your ISP tho. No 'edge-computing' here.
- Mobile orders. This will necessarily go thru to a central server. This is traditional client-server stuff. Again, no 'edge-compute'.
- Order receiving. Simple local data entry. Again, no 'edge-compute'.
- IoT devices. Hopefully these have local control systems without the server being in the loop. Control systems are not 'edge-compute' either.
'Edge-compute' is generation of knowledge at the edge rather than shipping raw data to a central server. This reduces required bandwidth.
What in your system takes a high rate of data and generates a low-rate of data for transfer to a server for further use. I see no analysis of raw data into a more processed form, this is simply traditional data entry and CRUD activites.
>In a recent post, we shared about how we do bare metal clustering for Kubernetes on-the-fly at the Edge in our restaurants. One of the most common (and best) questions we got was “but why?”. In the post we will answer that question. [emphasis mine]
Luckily there's a whole article on that topic. I try to avoid "Did you read the article?" comments, but the first three sentences in the article tell you that this is the question they're answering.
This is the relevant pair of paragraphs from the article:
>As a simple example, image a forecasting model that attempts to predict how many Waffle Fries (or replace with your favorite Chick-fil-A product) should be cooked over every minute of the day. The forecast is created by an analytics process running in the cloud that uses transaction-level sales data from many restaurants. This forecast can most certainly be produced with a little work. Unfortunately, it is not accurate enough to actually drive food production. Sales in Chick-fil-A restaurants are prone to many traffic spikes and are significantly affected by local events (traffic, sports, weather, etc.).
>However, if we were to collect data from our point-of-sale system’s keystrokes in real-time to understand current demand, add data from the fryers about work in progress inventory, and then micro-adjust the initial forecast in-restaurant, we would be able to get a much more accurate picture of what we should cook at any given moment. This data can then be used to give a much more intelligent display to a restaurant team member that is responsible for cooking fries (for example), or perhaps to drive cooking automation in the future.
I've enjoyed reading (all) your comments here. Just wanted to chime in and say I have a great appreciation for what CFA does and have had great experiences with the app and in general the entire ordering process at CFA. I've been, in fact, quite impressed. Nice to read about the stuff in the background that helps make that happen.
Doesn't hurt that every employee I've interacted with at CFA has been courteous, kind, and respectful.
Why pretend to be making a disinterested comment about how "kind and courteous" employees are (which is totally irrelevant) when you're actually motivated by sympathy to the company on the basis of their regressive, hate-fueled ideology? This kind of manipulation doesn't speak well of your fellow extremists.
I'd heard a few years ago about a random company called Chic-fil-a and that their founders were Christians and one of them said some things defending marriage that stirred up controversy. Then promptly forgot about them.
Then a couple years ago, one showed up near my work, and I heard they had good chicken. The first thing I noticed about CFA was how their employees acted. I was impressed! Then, I downloaded and installed the app and had a fantastic experience. I was impressed! And their food is great!
Certainly, I agree with the ideology of their founders.
However, that is independent of the fact that that I've been impressed! When I go to CFA, I wonder HOW they get me through the line so fast. I appreciate how they've trained their employees in good-old-fashioned southern hospitality. As far as I've seen, at this location, they hire friendly people who care about taking care of the customer.
So, back on topic... how DO they get me through the line so fast?
Hey this is Brian and I wrote the article. Great question.
1) MQTT -- we have a lot of cases where we want to share data between physical or "software" devices in-restaurant, without internet connections. MQTT is our primary channel for sharing messages between things, and for collecting data in general to be exfiltrated to the cloud.
2) "Brain" apps -- these are "smart" applications that collect data from MQTT topics and make decisions about what should be done for a given restaurant process. Today, the reality is we do not automatically cook anything. We do, however, have some restaurants that have more intelligent screens to help them know what to cook at any given moment based off of forecasting models that use the data we collect from many different things. These generally run at the edge to preserve their function in cases where there is high latency or bad WAN connections.
3) In the past we had a POS Server in the restaurant. It was a single point of failure for things like drive thru order taking with iPads and mobile orders. The move to K8s lets us run a microservice that can intake mobile orders on a more resilient infrastructure. To be clear, we have not made this shift just yet, but its something we are working on that will ultimately be transparent to customers but that will make their ordering experience more reliable. We have many cases like this.
Overall, our workloads aren't "heavy" which is why we did an array of three moderately sized commodity compute devices. We wanted a smart, sensible ROI on the hardware. We do have enough business-important apps that need to run in a low-latency high-availability environment to push us out of just cloud (internet dependent) and to the Edge. In the future, maybe this can change and we can use cloud only. It seems Edge is an industry trend though, with many workloads moving towards their consumers (a lot of gaming, but a lot of retail as well).
Great question and I hope this helped -- if not just let me know.
Good answer, seems we finally got to the meat of the product (no pun intended).
So the current workload doesn't do any of the fancy stuff but is just typical POS stuff with a 'smart' display for orders driven by a process monitoring a few MQTT sources.
You've just put in enough beef to allow for future expansion.
I can honestly say that answer provides more information than the whole original article.
Iot device control, such as timers for food, and cameras tracking food (to name a few). These also output streams of data that is interesting to us, such that we want to exfiltrate them up to the cloud.
I’m not a fan of hype... k8s is one of the few hyped technologies that has delivered on its promises.
Chickfila is very much a top-to-bottom ownership company. They own every step of the process in the cooking, from the machines, to servicing, to the logistics behind maintenance.
It's really exciting to see a company like this just want to own everything they do, and not reach out for 3rd party tools and SaaS and such.
They don't own everything, their logistics and servicing is through Trimark, granted many of those employees there only work on Chick-fil-a. But yeah Chickfila has always been one of the most high tech restaurant chains out there, its not common knowledge either. They have one of the most unique kitchen setups in the industry as well.
What data does CFA actually /need/ to collect in their restaurants? Orders and supply chain metadata? Store traffic statistics? Cleanliness and user ratings?
I hope it isn't Bluetooth or WiFi (or worse yet, GSM/LTE) metadata from their customers.
Our hypothesis: By making smarter kitchen equipment we can collect more data. By applying data to our restaurant, we can build more intelligent systems. By building more intelligent systems, we can better scale our business.
Yeah, I wish the article had more detail on the data and how they want to use it instead of the same overview of containerization you could find a thousand other places.
Somehow I missed the couple paragraphs mentioning kitchen equipment, and only saw the "Why does a restaurant company like Chick-fil-A want to deploy Kubernetes at the Edge?" bits. Those don't really give me any idea of what they're doing with the servers.
Thank you for the rude comment that doesn't even answer my question, though.
They didnt really cover it in the article, maybe in the video but I didnt watch it.
I've never worked for chickfila, but I've designed a number of restaurants kitchens before.
Chickfila has a large demand for using data driven solutions. Here are some common issues restaurants face:
1. How much food needs to be ordered? Excess food ends up being tossed out, sometimes it gets lost somewhere in the back of the walkin.
2. What days will be most busy based on major events happening in the area? How is this measured? How will we inform the manager about these changes so they can have more employees during this timeframe?
3. How are we going to know which food
4. How will we know when a piece of equipment needs maintenance, or if its done at all? (Grease traps, hood filter cleaning)
5. How are we going to track the customer experience?
6. How will we prevent cross contamination during preptime using camera systems?
7. During rush hour how can we ensure fries arent going to go stale and served effectively? E.g. computer vision. Where is that data going to be stored at?
8. If a customer has had a bad food experience how are we going to track the exact problem that happened?
9. How are we going ensure data redundancy when our ISP has internet outages? Whats the backup plan.
10. How can we ensure the filets were getting from sysco or whatever vendor is the right thickness tolerance? This will affect food temperate curve graph and quality of filet
11. How can we be sure rules are being enforced during rushhour, especially with new hires?
12. How are we going to measure effectivenesd of training protocols in chickfila and get feedback from staff for improvements?
13. How are we going to keep track of all the food inventory?
14. How are we keeping track of our parts distribution for service tech (JIT equipment?)
15. How are we gojng to automate the Profit and Loss support for managers?
16. How are we going to know our vegetable supply doesnt have ecoli? What bigdata metrics are we getting from vendors for this?
17. Why are customers given X chickfila store poor reviews?
18. How will we keep track of all the information used in each store(plans, equipment, etc)
Note corporate needs to keep track of the food quality being churned out at each store. It needs to be tracked... there is alot of data. And alot of systems to house and transfer that data
This is absurd. There is no scientific controversy regarding natural selection—there are only religious communities who willfully ignore the scientific method because it contradicts their foregone conclusions.
It's good to have the experience of interacting with someone far more intelligent than yourself who nevertheless believes things you think, with good reason, to be nonsense. It teaches one humility, to question the nature of "knowledge" and how it is attained, and the limits of "intelligence" alone to divine reality and arrive at consensus.
Sorry to interrupt the eager participation in this engineer recruiting effort (wow, iPads, cool!)
Don't forget that the people who run Chic-Fil-A support a toxic version of Christianity that, among other things, holds that God himself is going to punish the United States because I'm allowed to get married.
(Also, in case you're a few years ahead of the current western ethical consensus, they torture and kill a massive, genocidal number of chickens every year in order to better monetize related public health crises.)
Maybe don't work for them if you value modern society.
It's painful to scroll to the bottom and see all the suppressed comments about Chic-Fil-A's ethics. Hackers are powerful. Don't give yourselves over to people who would harm your society. If you make a business like this more profitable, you're literally putting money into the coffers of political organizations that seek to take away my rights.
