Page tree
Skip to end of metadata
Go to start of metadata

Rationale

The aim is to streamline the usage and maintenance of the plethora of Groovy scripts we have to carry out several system infrastructure tasks, from JIRA project creation to LDAP user management. 

The scripts are spread over different GIT projects under https://git.magnolia-cms.com/projects/SYS and work together by means of a complicated system made of Gradle build scripts. However, some scripts which should be of most common usage have been singled out, see Systools Script Cleanup.

Users of such scripts are both technical and sales people. This means a friendly UI would be better suited especially for the latter user typology. 

Problem

One of the ideas to streamline maintenance and usage was to convert the Groovy scripts into plain Java files. This turns out to be quite a big effort because

  • Each script typically depends on several other scripts which would need the same conversion. Basically one would end up rewriting the whole sys-tools in Java.
  • The conversion process isn't easy, due to reconstructing Java types where they're not specified (most scripts). You need to go through the source and figure out, for instance, what the various defs or untyped variables stand for. It's basically a painstaking manual process http://stackoverflow.com/questions/5302103/is-there-a-way-to-convert-groovy-to-java-automatically. An IntelliJ tool to convert Groovy source code to Java, though far from perfect according to some users, may still help to start the process and take some of the pain out of the task. 

Intermediate solution

A quick, ad interim solution (should there be the need for it) could be streamlining the usage of our sys-tools scripts via some sort of Java CLI. A library like https://commons.apache.org/proper/commons-cli/index.html could be used to provide some sort of inline help for the few scripts we want to keep.

The branches below are a quick PoC demonstrating how this could be achieved. The main ideas are

  • turn user-account-scripts into a Maven project proper so that its scripts can be imported and used by magnolia-sys-tools
  • build an executable magnolia-sys-tools uber-jar which will basically take care of packing all regular Java dependencies needed by the scripts, plus compile and package the Groovy scripts themselves (some of which are dependencies to other scripts).

https://git.magnolia-cms.com/users/fgrilli/repos/magnolia-sys-tools/commits?until=refs%2Fheads%2Fbugfix%2FDEV-327-implement-replacement-for-sys-tools

https://git.magnolia-cms.com/users/fgrilli/repos/user-account-scripts/browse?at=refs%2Fheads%2Fbugfix%2FDEV-327-implement-replacement-for-sys-tools

The end result would be invoking the scripts like this, for instance

$ java -jar magnolia-sys-tools-1.0.0-SNAPSHOT.jar GenerateMavenSupplementalModel /path/to/poms/build-resources/license-overrides.yaml /path/to/supplemental-model.xml

  • Advantages

    • Quick implementation. No complicated setup. No need for building anything (aside from running java -jar magnolia-sys-tools-1.0.0-SNAPSHOT.jar MakeConfig  once beforehand) or having Maven/Gradle/Groovy/Git installed. Just plain JRE will do.

  • Disadvantages

    • Some scripts which we don't need anymore would still stay around unnecessarily in our repos. Problem here is to determine exactly which scripts depend on which other scripts, a task that, with an untyped or weakly typed language like Groovy, may not be that trivial.
    • We'll keep on using Groovy which not all devs may be familiar with and does not have the same first class support from IDEs as Java. This may cause issues in case of maintenance or adding new functionality.  
    • We would still need to configure and maintain working version of scripts on laptops of each sales person (unless we plan to provide one place somewhere on the server where it would work and ask them to ssh to it)

As pointed out by Greg, we actually already have such executable jars which are built at https://jenkins.magnolia-cms.com/job/int_sys_user-account-scripts/ and https://jenkins.magnolia-cms.com/job/int_sys_tools/ (they're built with Gradle, instead of Maven, that's probably why I missed them while browsing through the sources).

Long-term (final) solution

Move away from Groovy and use Java for the whole sys-tools project. This would eventually constitute the back-end of an (intranet?) webapp with a user-friendly UI.

  • Advantages

    • Ease of maintenance and evolution. Best IDE support for debugging and refactoring. Clear view of dependencies between classes/modules.

    • Hassle-free for end users: they login to a webapp and are guided by a nice UI to accomplish their tasks.

  • Disadvantages

    • potentially long, tedious, error-prone process to clean up and convert all sys-tools Groovy scripts into Java.

Attempt at converting Groovy script conversion to Java - Lessons learned

I tried to convert one of the chosen scripts into Java /user-account-scripts/src/main/groovy/NewCustomerUser.groovy

The script depends on the following Groovy scripts/classes

  • info.magnolia.sys.tools.users.RegisteringUser
  • info.magnolia.sys.tools.users.UserRegistrar
  • info.magnolia.sys.tools.services.JIRA
  • info.magnolia.sys.tools.services.Ldap
  • info.magnolia.sys.tools.util.Util.*

the latter seems to be used mostly to interact with the user via terminal, e.g. prompt for data and choices, print info and errors.

I tried to create a basic Java UserManager interface

UserManager
public interface UserManager {
       RegisteringUser create(String email, String ldapSubOU, boolean hasSupport, SLAInfo slaInfo);
}

The implementation would delegate to a converted UserRegistrar to carry out the actual job of creating the user. A web app view controller of some sort would then use UserManager as a service.

Bad things

  • Code is messy, with lots of overloaded methods (often unused), commented out code and TODOs. “UI” and “backend” code are often mixed up making it even more confusing. For instance, one would think that UserRegistrar.groovy only deals with creating the user after getting the necessary data from NewCustomerUser.groovy. Instead one can find calls to UI methods which basically block the script execution flow and prompt the user to provide additional data
  • the scripts make use of closures which are not so obviously convertible into equivalent Java lambdas
  • the return type of some methods is very hard to guess unless going deep into the implementation until one finally gets to Java source code or some Groovy script along the way makes it more explicit
  • Using IntelliJ to convert to Java certainly helps in some cases but in others it only gets into the way. 
    • For instance this is how def jiraUID = null is translated into Java  final Reference<Object> jiraUID = new groovy.lang.Reference<Object>((Object) null). It takes some further investigation into the source to get to the correct final form String jiraUID = null;

Good things

  • apparently the bulk of the Groovy classes used as dependencies is under /magnolia-sys-admin-tools/src/main/groovy/info.magnolia.sys/tools/services and /user-account-scripts/src/main/groovy/info/magnolia/sys/tools/users
    • It’s about 30 classes but, at a first glance, JIRA.groovy and Ldap.groovy are the most used ones.
    • although not a trivial nor pleasant task, conversion to Java looks feasible, even though it's going to take some time.

 

DevOps meeting 13.12.2016

Participants: 

momo

Pat Hills

Bradley Andersen

Philip Mundt 

Federico Grilli

Summary:

The options presented in this page were discussed. Alternative solutions were proposed as well. No decision made.

LDAP management scripts

The scripts have been partially replaced with https://magdap.magnolia-cms.com/ and some custom Bash script operated by DevOps.

SLA info are going to be managed by LDAP only, so no need to interact with JIRA anymore.  A tool with a friendly UI would be basically needed by sales only to create new customers/partners.

All other scripts

Have lesser priority. We can live with current Groovy scripts also run via executable jars. 

Pro/Contra summary table

OptionProContraNotes

Use executable jars

We already have them. Quickest solution.

Maintenance issues

Suitable for some scripts used only by tech people (e.g. GenerateMavenSupplementalModel)

Place them on some server and allow users to ssh to it and run something like java -jar MagnoliaSysTools.jar SomeScript

Convert scripts to Java

Feasible if we don’t have to convert ALL the scripts

Conversion kind of painful, will take some time, even if limited to a few scripts 

Rewrite in Java from scratch

Cleanest solution, we focus only on what we need

Requires a good knowledge of JIRA and LDAP and their setup at Magnolia which may turn it into an even harder task than converting the scripts 

Suggested by Greg

Rewrite as Content AppWe leverage Magnolia existing infrastructure.
Fits the master/detail app paradigm, which would suit well the LDAP use case.
Magnolia employees should be familiar with apps.

Would only solve the LDAP user management case. Not suited for other sys-tools operations.

Seems overkill to setup one Magnolia instance for just one app.

Suggested by Bradley.

Buy a dedicated tool

We don't need to maintain it.Would basically solve the LDAP use case only.
May be too complex for non tech users.  

Suggested by Greg. Preferred option by DevOps.


 

 

  • No labels

2 Comments

  1. Hey Federico - quite frankly if you want to do this in Java, you're probably better off rewriting than trying to "convert". Those scripts have evolved...very organically. Also, if your plan is to make a web app out of it, definitely rewrite. Or, probably more adequately, buy the proper tool for the job. (I did not find one at the time, but, times have changes). Also those scripts do 374 things you'll never want a web app to do (smile)

    1. That's actually one option I thought of but looks like an even more daunting task than converting. Let's see (smile)