The aim is to streamline the usage and maintenance of the plethora of Groovy scripts we have to carry out several system infrastructure tasks, from JIRA project creation to LDAP user management.
The scripts are spread over different GIT projects under https://git.magnolia-cms.com/projects/SYS and work together by means of a complicated system made of Gradle build scripts. However, some scripts which should be of most common usage have been singled out, see Systools Script Cleanup.
Users of such scripts are both technical and sales people. This means a friendly UI would be better suited especially for the latter user typology.
One of the ideas to streamline maintenance and usage was to convert the Groovy scripts into plain Java files. This turns out to be quite a big effort because
- Each script typically depends on several other scripts which would need the same conversion. Basically one would end up rewriting the whole sys-tools in Java.
- The conversion process isn't easy, due to reconstructing Java types where they're not specified (most scripts). You need to go through the source and figure out, for instance, what the various
defs or untyped variables stand for. It's basically a painstaking manual process http://stackoverflow.com/questions/5302103/is-there-a-way-to-convert-groovy-to-java-automatically. An IntelliJ tool to convert Groovy source code to Java, though far from perfect according to some users, may still help to start the process and take some of the pain out of the task.
A quick, ad interim solution (should there be the need for it) could be streamlining the usage of our sys-tools scripts via some sort of Java CLI. A library like https://commons.apache.org/proper/commons-cli/index.html could be used to provide some sort of inline help for the few scripts we want to keep.
The branches below are a quick PoC demonstrating how this could be achieved. The main ideas are
- turn user-account-scripts into a Maven project proper so that its scripts can be imported and used by magnolia-sys-tools
- build an executable magnolia-sys-tools uber-jar which will basically take care of packing all regular Java dependencies needed by the scripts, plus compile and package the Groovy scripts themselves (some of which are dependencies to other scripts).
The end result would be invoking the scripts like this, for instance
$ java -jar magnolia-sys-tools-1.0.0-SNAPSHOT.jar GenerateMavenSupplementalModel /path/to/poms/build-resources/license-overrides.yaml /path/to/supplemental-model.xml
Quick implementation. No complicated setup. No need for building anything (aside from running
java -jar magnolia-sys-tools-1.0.0-SNAPSHOT.jar MakeConfigonce beforehand) or having
Maven/Gradle/Groovy/Gitinstalled. Just plain
- Some scripts which we don't need anymore would still stay around unnecessarily in our repos. Problem here is to determine exactly which scripts depend on which other scripts, a task that, with an untyped or weakly typed language like Groovy, may not be that trivial.
- We'll keep on using Groovy which not all devs may be familiar with and does not have the same first class support from IDEs as Java. This may cause issues in case of maintenance or adding new functionality.
- We would still need to configure and maintain working version of scripts on laptops of each sales person (unless we plan to provide one place somewhere on the server where it would work and ask them to ssh to it)
As pointed out by Greg, we actually already have such executable jars which are built at https://jenkins.magnolia-cms.com/job/int_sys_user-account-scripts/ and https://jenkins.magnolia-cms.com/job/int_sys_tools/ (they're built with
Gradle, instead of
Maven, that's probably why I missed them while browsing through the sources).
Long-term (final) solution
Move away from Groovy and use Java for the whole sys-tools project. This would eventually constitute the back-end of an (intranet?) webapp with a user-friendly UI.
Ease of maintenance and evolution. Best IDE support for debugging and refactoring. Clear view of dependencies between classes/modules.
Hassle-free for end users: they login to a webapp and are guided by a nice UI to accomplish their tasks.
potentially long, tedious, error-prone process to clean up and convert all sys-tools Groovy scripts into Java.
Attempt at converting Groovy script conversion to Java - Lessons learned
I tried to convert one of the chosen scripts into Java
The script depends on the following Groovy scripts/classes
the latter seems to be used mostly to interact with the user via terminal, e.g. prompt for data and choices, print info and errors.
I tried to create a basic Java UserManager interface
The implementation would delegate to a converted
UserRegistrar to carry out the actual job of creating the user. A web app view controller of some sort would then use
UserManager as a service.
- Code is messy, with lots of overloaded methods (often unused), commented out code and TODOs. “UI” and “backend” code are often mixed up making it even more confusing. For instance, one would think that
UserRegistrar.groovyonly deals with creating the user after getting the necessary data from
NewCustomerUser.groovy. Instead one can find calls to UI methods which basically block the script execution flow and prompt the user to provide additional data
- the scripts make use of closures which are not so obviously convertible into equivalent Java lambdas
- the return type of some methods is very hard to guess unless going deep into the implementation until one finally gets to Java source code or some Groovy script along the way makes it more explicit
- Using IntelliJ to convert to Java certainly helps in some cases but in others it only gets into the way.
- For instance this is how
def jiraUID = nullis translated into Java
final Reference<Object> jiraUID = new groovy.lang.Reference<Object>((Object) null).It takes some further investigation into the source to get to the correct final form
String jiraUID = null;
- For instance this is how
- apparently the bulk of the Groovy classes used as dependencies is under
- It’s about 30 classes but, at a first glance,
Ldap.groovyare the most used ones.
- although not a trivial nor pleasant task, conversion to Java looks feasible, even though it's going to take some time.
- It’s about 30 classes but, at a first glance,
DevOps meeting 13.12.2016
The options presented in this page were discussed. Alternative solutions were proposed as well. No decision made.
LDAP management scripts
The scripts have been partially replaced with https://magdap.magnolia-cms.com/ and some custom Bash script operated by DevOps.
SLA info are going to be managed by LDAP only, so no need to interact with JIRA anymore. A tool with a friendly UI would be basically needed by sales only to create new customers/partners.
All other scripts
Have lesser priority. We can live with current Groovy scripts also run via executable jars.
Pro/Contra summary table
Use executable jars
We already have them. Quickest solution.
Suitable for some scripts used only by tech people (e.g.
Place them on some server and allow users to ssh to it and run something like
Convert scripts to Java
Feasible if we don’t have to convert ALL the scripts
|Conversion kind of painful, will take some time, even if limited to a few scripts|
Rewrite in Java from scratch
|Cleanest solution, we focus only on what we need|
Requires a good knowledge of JIRA and LDAP and their setup at Magnolia which may turn it into an even harder task than converting the scripts
Suggested by Greg
|Rewrite as Content App||We leverage Magnolia existing infrastructure.|
Fits the master/detail app paradigm, which would suit well the LDAP use case.
Magnolia employees should be familiar with apps.
Would only solve the LDAP user management case. Not suited for other sys-tools operations.
Seems overkill to setup one Magnolia instance for just one app.
|Suggested by Bradley.|
Buy a dedicated tool
|We don't need to maintain it.||Would basically solve the LDAP use case only.|
May be too complex for non tech users.
Suggested by Greg. Preferred option by DevOps.