I treat small data like big data. It’s not their fault they’re small and it doesn’t mean they can’t live like their beefy cousins.
I work exclusively with Azure tech so it’s safe to assume there should be an “Azure” affixed to most of the products I use.
If you want the less decipherable version, I design and build an ETL system using Data Factory, ADLS Gen2, Databricks, and Cosmos DB as the core components. I hide some stuff in Key Vault and I upset people who know about DevOps by teaching myself the mysteries of Pipelines, ARM templates, and Blueprints.
I often reinvent the wheel or I build it with the wrong tools without realizing it. But I’m not afraid to take my lumps. I threw away months of U-SQL work, saw the beauty of Databricks, and rebuilt my system from the ground up in a few weeks. That really reaffirmed that learning the tools should be a small part of the problem. Knowing what to do with the tools is the challenge.
I also use a lot of words to say little things. I’m here to figure out which of those words are the important ones so I can throw out the others.