I inherited a data pipeline with a daunting amount of infrastructure, code, database artefacts, and huge volumes of logs.

I found that the CICD pipeline was creating artefacts for each run...

The whole world runs on hidden heroes in all endeavours.  An awful lot of stuff works because someone quietly did a thankless task, sometimes in the face of hostility from...

I am using Snowflake, which has several timestamp data types.  It's a legacy codebase, so usage across the codebase is a bit random.

• TIMESTAMP_LTZ  stores UTC, but any interaction will...

I'm a Principal Data Engineer.  One of the things I do is to look at what other principals are doing in their areas and weigh the pros and cons of...

Brent's point is backed up by the theory of constraints.  The theory is that you only have one critical bottleneck constraint at a time.

The more impressive motor racing feat is...

This is a case of "The future is already here—it's just not evenly distributed".

Over the past couple of decades, I've seen shutdowns for major releases decline as architectural and software...

All the work I do is in the cloud, and the Database As A Service model means that patching and upgrading are mostly outside of my jurisdiction.  I can choose...

I knew about GROLIES (Guardian Reader Of Limited Intelligence...etc) and...

What works for me is the Disney method.  Write the ideas down, don't critique or polish at the idea stage (dreamer stage).  I was introduced to the Disney method...

Back in 2017, I would have agreed with this 100%, but there are edge cases.  There's the joke about 2 senior managers talking about their DBAs.  One says, "My DBAs...

I've found that there are tests and there are well-thought-out tests.  If tests police your development, then they need to be held to the highest standard.  Quis custodiet ipsos custodes? ...

Steve, I love that sentence, "Learn to be a team player as an engineer, developer." I see a lot of people who aren't team players. However, in a real...

• This reply was modified 6 months, 3 weeks ago by David.Poole.

These lessons are all great.

One lesson I would add is to make sure that the right people are acknowledged for their work.  As a senior, it is great to be...

I ceased to be a DBA 10 years ago when I took a chance on getting into Big Data.  I think regret is too strong a word, but I often...

I read the article about the cost of software development approaching zero and realized the author was talking about one off solutions of a small and specialized nature.  I...