Skip to main content

Office Hour Topics

Data Formats & Tools​

  • Writing Parquet files to S3
  • Parquet in practice (fastparquet vs pyarrow, performance, time series use)
  • How to stop terminal from hanging when saving Parquet files
  • awswrangler - what it does and how to use it

Data Storage & Architecture​

  • Data lake - what it is, how it works, why it’s used
  • Data warehouse vs database
  • OLTP vs OLAP
  • Data modelling approaches (1NF/2NF/3NF, relational models, star schemas, etc.)

SQL Theory & Engines​

  • SQL dialects - why so many variants instead of one standard
  • Query optimisation
  • Advantages and disadvantages of different SQL engines
  • Relational algebra

AWS Services & Cloud Data​

  • AWS Glue - what it does and why Athena can’t query S3 directly
  • Logging with CloudWatch (special considerations for deployed instances)
  • Recap on AWS networking (subnets, subnet groups, security groups, VPCs)

Containers & Virtualisation​

  • ECS vs ECR
  • When to use EKS vs ECS (and Kubernetes differences)
  • Tasks vs Services in EC2
  • When to host on a virtual machine vs container
  • Docker and OS virtualisation (what is a …)