Hacker News new | past | comments | ask | show | jobs | submit login

Looker on top of Redshift. Events streamed in from Segment and ELT transforms managed by Airflow. Looker gives you nice visualizations and charting with RBAC and some some lightweight ETL functionality. The real advantage of Looker is their modeling layer which sits on top of your physical data and is done via a DSL called LookML. Source control is managed via a wrapper around git. The end result is that analysts can move lightning fast building their own models in SQL or do analysis via drag and drop explorations. Looker's customer support is the best I've experienced and hasn't changed since Google acquired Looker. We're likely moving off Redshift to Snowflake in the next 6 months because it is slow to pick up changes and we want to decouple storage and compute for scaling reasons. Airflow is an incredible platform but needs customization and templated workflows to make it great. Data build tool (DBT) is how we plan on managing ETL code via Airflow in the near future. We're also adding Spark, but more for data science.



If you're interested in an alternative to dbt that's a little more analyst friendly, check out Dataform. A lot of teams are using Dataform between the warehouse and looker, to handle transformation logic using pure SQL, but in a more user friendly format than Airflow. Get in touch if you'd like to chat! dan at dataform.co




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: