Devops roles in the project

[leshikus]

We have discussed yesterday that communication between devops and development should become more formal, here is a list of things which devops (me) did in the past as requested

1. Scripting (bash + github + cmake + dockerfile) - this is an easy part for me; it takes little time for to debug a problem in my script and it takes little mental effort; it is sometimes time consuming, yet I can address it by adding some debug infrastructure to my scripts myself. It is rewarding and creative.

1.1 Moving old / new / outdated processes to github actions
1.2 Updating the processes when an environment changes, e.g. ubuntu ships a new gcc as default one, it comes to github and our build which relies on Ubuntu disto fails

I can easily continue supporting the changes in scripts assuming the build itself works. I think though we can all benefit from knowledge sharing with respect to cmake or other things.
 
2. Resolving problems / troubleshooting
This is hard and time consuming.

2.1. Debugging reliablity problems - e.g. out of memory, zombie ray processes, cache problems etc.
2.2. Debugging environment problems - e.g. cuda or conda updates make the build to fail.
2.3. Debugging new hdk issues like ASAN build stopped working because it stopped fitting into managed agent memory.
2.4. Working as Linux server admin, e.g. updating packages and communicating if I can reboot the server afterwards.

3. Cleaning up existing code
   This is not time consuming, yet keeping the code bloadted and unhealthy makes it more difficult to debug, see item 2. For example, if our conda env has a minimum number of required packages listed, than it is much easier to find a conflict. (It does not.) The code also needs to be updated because environment and requirements change. (Example: using pre-built artifacts is a common CI practice and Intel strives to follow best CI practices.)

As for the items 2, 3 I get feedbacks that I spent too much time on them with little result. This makes me unsure if I'm things correctly here