advice on diving into devops for managing BI projects
-
hi , i know this is a sql server forum but i think my brain is aligned more with members of this forum than any other.
we have a large netsuite migration in which multiple erps are involved. The technologies on which those erp's run vary from db2 to all flavors of sql server based products. Our team's interest lies in how those projects affect our warehouse. Each migration, in my mind, comes down to a dw conversion (for existing dw history dimensions) and extract (etl) re engineering.
in my head, we would have an Epic for dw impact overall. And Features representing the roadmap for each erp. Stories would represent conversion and extract changes separately. Tasks would be that detail we all know so well. From my experience, repeating key words in the names of these hierarchical layers is important in recognizing what something means if you dont have the benefit of seeing parentage/context in the view you are using.
Its been a while for me since i used devops for BI related project oversight. Does the community have any advice?
Right now my peers and i struggle a bit envisioning how any testing that is at a higher level than unit testing would be represented. i suppose sub tasks can help here and maybe stories that are themselves about testing. But what if the impact between Features needs to be system tested. Im having a mental block on that. And from what i remember, at least once i had to resort to excel instead of devops to manage a longer list of related tasks that were difficult to keep straight in devops. Also, we may want to create hooks into service now from devops but the jury is out on that one. Last time i used devops, our stories were written using a language format that has a certain name. If you recall a song called "if i had a hammer..." it kind of reminded me of that cadence where the story might start with something like "if i was a financial analyst ...." and would follow with what the story must accomplish from the perspective of the financial analyst. it was weird but somewhat effective.
-
My opinion - I think it really depends on your team and how you use your tools but it sounds like you are talking about Jira specifically (I could be wrong, but it sounds like Jira terminology). If it is Jira, then my approach is a little different than yours. I would have a project which would be for the DW impact that would be something like "DW Migration" as it sounds like this is a repeated process, so I would reuse the project for each migration.
I would have epics to define a release. When the epic is complete, the changes are released to live. This doesn't mean the project is "complete", it just means a major milestone was reached. If your project is to migrate everything in 1 go, then the epic would be to migrate the entire DW. BUT if you are able to migrate 1 ERP at a time, then I would have 1 epic per ERP. Whatever makes sense for a release.
Features I wouldn't actually use in an ERP based project, but that is your call. I tend to use features more for application level projects where we are tracking the required application features that must be completed before the epic can go live.
Stories is where I would put the generalized steps. This would include the ERP migration, the ETL migration, new servers if applicable, etc. Then under those I would have tasks that define what needs to be done in general terms if applicable. For example, I may make several ETL migration stories if I can see a way to group the ETL jobs or I may just have 1 general story and then 1 task per ETL job IF I am going to assign them to multiple people. If a story doesn't need a task, then I would just have a story. For example, stories that are applicable outside of your department (new server for example), the setup steps are out of your control. The infrastructure team MAY have their own checklist or system for a new server. Anything outside my control I leave as a story, assign to the project lead, and have them put in a support request to the infrastructure team to handle.
Since the stories are tied to the steps, tasks can be created under the stories for testing or depending on when you plan to do testing, you may want testing to be its own story. If testing is happening on each story as the story is complete, then I'd make it a task of the story. If testing is only happening once the release is ready, I'd have a story for testing and then add bugs to the stories that are problematic.
In my opinion, if you have too many tasks, you should see what you can do to generalize the tasks and combine some. For a very bad example, I wouldn't have a task to open visual studio and another to load up the ssis package and another to modify feature A and another to modify feature B and so on. I would have one like "update SSIS packages related to <process A> to work with new DW". This may end up being 1 SSIS package or it may be 10 or 100. If it is 100 though, I'd look into breaking that up into more consumable tasks. You can also make checklists on a task, so you could still list all packages that need to be modified and then assign the task to someone to work on it and they can mark off the checklist as they complete different packages.
The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system. -
thx brian, i'll reread this tomorrow. i thought jira and devops were 2 different things. and either way dont remember a level in devops above epic called project. that would definitely change things. got some reading and collab to do.
-
Perhaps the word 'DevOps' has confused things here? I'm assuming you're talking about 'Azure DevOps' specifically, and not DevOps is general?
Azure DevOps can be used for several things, including task management via the backlog/sprint functionality (which is what I think Stan may be referring to).
But the term DevOps is perhaps more commonly associated with continuous integration, via source control, build and release capabilities (which is perhaps what Brian was thinking of).
The epic/story/feature stuff is similar between the two, with Jira being probably the more mature platform.
Therefore, I think that Brian's comments around managing the project in Jira can also be applied to Azure DevOps.
-
Phil Parkin is right about what I was thinking of - I was thinking more of the TERM DevOps and initially thought of CI/CD and was thinking this was going to be a git related question but then read the post and terminology sounded a lot like what Jira does. Didn't actually know about Azure DevOps so my advice is probably a bit off due to the different tool.
Jira has "projects" where you track the epic's, stories, features, bugs, tasks, subtasks, etc. and you set up workflows around that to control releases. You can even create custom issue types if you don't like the default ones so you could have things like "Nice To Have Feature" and "Required Feature" so when you work on the project you complete all the required features then can tackle the nice to have features if time permits. If it doesn't those get put onto the roadmap.
It's fun when multiple tools use the same terminology to mean different things, just might need to tweak the advice to suit the tool.
Jira is nice but expensive. The core functionality is there to get you past the basic stuff, but then you find functionality that you want that is lacking. That's where the marketplace comes into play and before you know it, you are spending 500+K per year on the silly tool.
The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system. -
Thx Phil. You are correct.
Ok Brian...
I checked and the "off the shelf" level in azure devops that correlates to your jira project level does exist but will represent our team's root or "space" in this technology. It will simply be called "BI" or "BI Team". I feel pretty comfortable with that decision. My Epic level would then correlate to your project level as where either of us chooses as the top of the software's usage hierarchy.
So if we had to do this in parallel with the way you do it, and my proposal was to be discussed first, your Project (our Epic) would be "NetSuite Migrations". Switching to your approach, your preference would be that our Epic be called (something like) "DW Migrations". I think both fit the definition (at least AI's) of an Epic being "a large body of work or high level goal that cant be completed in one sprint, spanning multiple features and potentially months".
Moving on, you'd make our feature level (akin to your epic) releases, like "ERP X Historic DW NetSuite Data Conversion" and "ERP X DW ETL change for NetSuite". I think that also fits the definition "A significant piece of functionality delivering value, broken down from an epic, often taking 1-3 sprints" except maybe for the 1-3 sprints part. I understand you dont like using features for non app projects but we'd be somewhat forced into using this level if release "ables" need to be delineated.
Then, i think we both arrive at the story level. And if appropriate, tasks and checklists below. Practice will tell us if your vision is best for us. At this moment, your release "ables" appear to also be stories. We'll see.
As for testing, i think you make it more palatable. There is no reason that if a certain testing effort is more like system testing whose scope also includes a previous story or feature (regression), that it just be an additional story that states that regardless of where it sits hierarchically.
I'll pass this on to our team. Do you remember what that "language" is called for writing stories? Its not what everybody uses but it has a distinct cadence/structure. it tended to get complicated because of all the scenarios (use cases) you'd have to list.
-
it took a minute for the light to go on but the word Feature sounds more like a releasable than the word story. That bums me out a little. my stories are releasable(s).
I'll have to look at what our other teams are doing.
In the meantime, this is that language i was asking about. At the moment it looks like its called the "standard user story format".
-
i've asked the folks who set this up if we can move "BI Team" to a higher level (maybe there is an ORG or OWNER?) and use Project (the what and why) differently. will post back here.
Viewing 8 posts - 1 through 8 (of 8 total)
You must be logged in to reply to this topic. Login to reply