Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Ask HN: Learning material for data design/architecture
6 points by StefanWestfal on Nov 24, 2022 | hide | past | favorite | 3 comments
I worked for a large cooperation where senior engs. would design the overall layout of our services, and we (junior) focused on implementation. Now I work at a startup and am involved design decisions. Therefore, I would like to learn about software design and architecture. About the application, we are medium-sized company and in a range where computations could run on a local machine (no-big data) but the data themselves can vary a bit from customer to customer.


IoT stuff is the easiest stuff to understand when it comes to high-volume data processing because it's a wide problem (ie: you have a whole lot of devices sending data continuously) instead of a deep problem (you have a few sources sending lots of data continuously).

IoT tends to talk more about HA (high availability) etc than more 'normal' architecture books, because it's about practical solutions.

I can't think of any right now (I looked on my bookshelf and it looks like the book was lent out). But any IoT architecture book will give you the basics.

Then maybe go look at the introductions to storm/kafka/spark.


One thing to remember that's surprisingly important is to understand how the upstream people are using your data. The shape of data in your datastore should match what upstream the consumers need.

The more it matches their use cases the less work it'll be to transform the data. Transformation == work == time.


Some good starting points:

- Designing Data-Intensive Applications, Martin Kleppmann

- Building Microservices: Designing Fine-Grained Systems, Sam Newman

- Domain-Driven Design, Eric Evans




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: