Product Development

Page tree
Skip to end of metadata
  • Created on 2016-03-17, last edited by Bradley Andersen on 2016-05-04
Go to start of metadata

 

DEV-159 - Getting issue details... STATUS

DEV-178 - Getting issue details... STATUS

 

Update

The POC setup tools has become a new Magnolia project. Here are its coordinates

Jira:  https://git.magnolia-cms.com/scm/build/npm-jumpstart.git

GIT: https://git.magnolia-cms.com/projects/BUILD/repos/npm-jumpstart/browse (master [jetty+Magnolia web app ] and tomcat-bundle branches)

Jenkins: https://jenkins.magnolia-cms.com/job/build_npm-jumpstart/

 

 

Inspired by Tomáš's presentation at Lunch&Learn about Node.js and its package manager npmjs based module called CLI https://git.magnolia-cms.com/users/tgregovsky/repos/cli/browse.

Rationale

Since enabling support for resource loading and yaml-based configuration, Magnolia has given front end developers extra tools for the simple creation of templates and tools for providing end users with updates in design and templates without the need for re-deployment. The combination of template definitions, templates and other resources created using this new offering is called Magnolia Light Modules (LM).
The recent introduction of NPM as a means for packaging and distribution of LMs additionally strengthens this offering by giving front-end developers tools for obtaining and distribution of such LMs.  This encourages sharing them within different projects or even different teams.
While the enabling of resource loading and configuration via yaml has allowed front-end developers to use their favourite tools when working with Magnolia and NPM based tooling greatly simplified distribution of results of their work, the problem remains (same as previously with JCR based configuration) with the number of points at which even an experienced user might make mistakes by either placing configuration at the wrong location, unintentionally omitting some mandatory structures, or simply misspelling names of required structures in configuration files.
Aside from that, the wide range of configuration parameters offered by Magnolia makes it difficult for front-end developers to remember all possibilities, forcing them to frequently consult documentation in order to be able to proceed with template definition work.

Goals

Provide an npm package for front-end devs to get started fast and develop painlessly (hopefully) with Magnolia and light development.

More precisely, provide command line based tooling (CLI) for front-end developers that can be used to automate common tasks, reducing the number of possible error points, as well as removing repetition and allowing front-end developers to concentrate on their goals: building websites that are visually stunning and easy to use.

Benefits

  • Increased efficiency by automation of daily tasks (creation of page and component templates, assignment of components to pages, creation of dialogs associated with pages or components)
  • Reduced number of possible errors (by generating common structures automatically, limiting possible errors to misspellings of component names or dialog fields added by user, but ensuring that structure is correct)
  • Lowered barriers of entry for becoming front-end developers using Magnolia (learning a couple of commands to generate basic website structure vs. learning the whole configuration structure for template definitions just to create first templates)
  • Additional, secondary benefits are
    • Further increases in development efficiencies due to reduced number of testing configuration cycles (due to lower number of errors / regressions)
    • Learning advanced configuration is simpler (by allowing front-end developers to inspect generated structures)
    • Ensuring better understanding of custom work within project teams (generated structures are always same no matter which developer generated them; encourage people to place additions at the same locations)

Constraints

  • Whatever structures and placements for the resulting artifacts are generated by scripts need to be aligned with the structures generated by npm magnolia-build & distribution scripts
  • Scripts need to operate correctly even when one chooses to not use npm for distribution (e.g., for small and internal projects)

Delivery timeframe

  • Final version should be released, at the latest, with Magnolia 5.5.
  • Alpha version for the 2016 conference in Basel.

Solution

Since Magnolia already supports npm as a tool to build and distribute LMs, automated tasks can and should be provided by additional npm-based scripts. 
Scripts for downloading and starting Magnolia distributions are already in the jumpstart project.
Prototype of remaining functions (and scripts) already exists (taken from original contribution by Tomáš) and is analyzed as part of jira@DEV-178.

Scripts described below can be either installed as global and executed without need for npm run prefix, or just locally with it. In the case of local execution only, one needs to have scripts defined in one's package.json file.

Jumpstart

Functions supported by jumpstart
  • npm start to download and start Magnolia initially
  • start-magnolia
  • stop-magnolia
  • restart-magnolia

As part of start operation, magnolia.properties in downloaded and extracted instances are also adapted.

It is possible to point scripts to existing Magnolia installations, and to start such existing installations (for users with custom locally-only available bundles).

Configuration and use of CE or EE instance of Magnolia is also possible by modification of the package.json file.

CLI

Supported operations
  • Creation of pages; e.g., add-page myHome
  • Creation of components;  e.g., add-component myImage
  • Adding components availability to pages;  e.g., add-component myImage available@sampleModule:pages/myHome@main
  • Adding auto-generated components to pages;  e.g., add-component footer autogenerate@sampleModule:pages/myHome@footer
Scripts allow for configuring availability and auto-generation only for page templates defined inside of the current light module/project; however components made available can come from anywhere (other light modules or other classic modules). No validation of errors against live instance of Magnolia is performed.

Future ideas

  • A custom command might allow users to download and place additional Magnolia modules automatically in WEB-INF/lib (resolving dependencies in that case could be tricky, or we need to limit this to downloading bundles that have all dependencies already packed).

  • tl;dr parameters for experts:
    • availability: create-component textImage -a pages/myHome@main
    • autogenerate: create-component textImage -g pages/myHome@main

Decisions

  • include jumpstart part of the project
  • do not use any other tools such as yeoman or gulp or others to avoid extra complexity
  • none of the proposed solutions for install are ideal; explore yet another possibility. The one used by Tomáš is closer to how other tools install (e.g. gulp) and should be preferred in absence of better ideas
  • use create-page and create-component instead of add- 
  • add-availability will stay as is
  • syntax change for parameters - use --available and --autogenerate instead of available@
  • module name is optional (if not provided, current is assumed, when provided we look for other project on the same level where current project is (in the same folder), when not found, simply fail with reasonable message)
  • examples of updated syntax:
    • create-component image
      • Add/create component template (will create component yaml file, component ftl and component dialog) 
    • create-component textImage --available [sampleModule:]pages/myHome@main
      • Same as above plus will set this component available on 'myHome' page in 'main' area (if area or availability configuration doesn't exist's in 'myHome' page, will create it as well) 
    • create-component footer --autogenerate [sampleModule:]pages/myHome@footer
      • Same as above, just will do 'autogeneration' instead of availability
    • add-availability [mtk:components/]html [sampleModule:]pages/myHome@main
      • Make component 'html' available at 'myHome' in 'main' area
    • create-light-module [-p path/to/light-modules-root-folder] myModule
      • Create a new light module structure with the standard Magnolia light dev structure

Outstanding

Items below require additional research and experimentation or discussion or decision or all of those
  • Which bundle to use for jumpstart custom jetty or standard tomcat?

Next Steps

  • Exploratory task for better install of cli  DEV-193 - Getting issue details... STATUS
  • Refactoring for new syntax  NPMCLI-8 - Getting issue details... STATUS
  • Refactoring to remove need for use of "sudo"  NPMCLI-5 - Getting issue details... STATUS
  • Refactoring to remove need for YAML->JSON->YAML conversion  NPMCLI-4 - Getting issue details... STATUS
-----------------------------

 

Experimenting with Node.js and npm

 

As I'm a newbie to Node.js I had first to get familiar with the new technology. The result of my trials and errors is the little module at https://git.magnolia-cms.com/users/fgrilli/repos/mgnl-light-dev-poc/browse  https://git.magnolia-cms.com/projects/BUILD/repos/npm-jumpstart/browse

There are two flavours of the module on two different branches 

  • master: a module using Jetty + Magnolia community web app 
  • tomcat-bundle: a module using Magnolia's CE tomcat bundle

From the module's readme.txt

This is a PoC for a nodejs module (package) which installs and runs a Magnolia instance ready for light development.
...
Run:
- run "npm start". This will
-- connect to Magnolia's public repository on Nexus and download Magnolia (see package.json)
-- extract Magnolia
-- change some magnolia.properties (again see package.json)
-- start up Magnolia

Custom commands:
- Apart from the standard npm commands corresponding to the phases of a package lifecycle, e.g. install, start, etc.
this module provides these additional commands 
-- start-magnolia, usage "npm run start-magnolia" 
-- stop-magnolia, usage "npm run stop-magnolia"  
-- restart-magnolia, usage "npm run restart-magnolia"


 

21 Comments

  1. Christopher Zimmermann

    Wow - sounds great! Look forwards to testing this out.

  2. Christopher Zimmermann

    OK - that was a very cool magnolia starting experience.

    Two notes:

    1. In the readme: 

    - run "npm install --save"
    -- This will install the module dependencies from npmjs under node_modules folder in the current directory.

    Actually you just need to run "npm install", the "–save" is for adding a new module to the package.json- which you are not doing here.

    2. The progress indicator thing on download was not a nice experience - the entire console was filled with tons of downloading updates - is there a way to get that to just update one line in the console? Or is that some local issue on my machine?

     

    1. Federico Grilli

      - run "npm install --save"
      -- This will install the module dependencies from npmjs under node_modules folder in the current directory.

      Actually you just need to run "npm install", the "–save" is for adding a new module to the package.json- which you are not doing here.

      Thanks, will fix it.

      The progress indicator thing on download was not a nice experience

       I experienced that a couple of times as well. I think it may have to do with either resizing the terminal window and/or terminal character size. Either case I agree it should be fixed so that it looks nice regardless of user's settings.  

      1. Christopher Zimmermann

        I wonder how npm does their progress bar? That seems to work well.

  3. Christopher Zimmermann

    I think it is worth exploring yeoman. I think its a good idea to invest some time there.
    But we should also keep in mind to balance what benefits it brings, vs the extra complexity of another dependency. 
    I like the idea of using it. It is something many front-end devs are used to. But we need to consider the overall simplicity too - not just what front-end devs like now. 

  4. Christopher Zimmermann

    I the jetty branch, I like how the js files are moved into bin.

    I did not find much of an improvement in total install and setup time:

    magnolia tomcat + demo + author + public: 2:12 min
    magnolia jetty: 2:14 

    After install, startup times were the same (once i removed the Public from the tomcat intance.) 19 seconds.

    As you mentioned in the readme - the failure and restart with Jetty would have to be fixed. 

     

  5. Christopher Zimmermann

    I like that Jetty shuts down on CTRL-C.

    I would be hesitant to use Jetty instead of Tomcat simply because its another thing to explain and document, especially if they behave differently.

    Its worth considering- but there must be clear advantages that would outweigh the cost of explaing and potential confusion of devs. 

    1. Federico Grilli

      I'll try to come up with a list of cons and pros about the two approaches asap

  6. Jordie Diepeveen

    Amazing POC! Really nice to setup a light-module environment in a few seconds.

    Hmm. it seems that the light-modules folder is created one level higher then the folder containing the package.json file, so outside the project folder.

  7. Federico Grilli

    An alternative syntax for commands in the style of Git subcommands can be implemented fairly easily with the commander package https://www.npmjs.com/package/commander#git-style-sub-commands (I already tried it locally starting from this branch https://git.magnolia-cms.com/projects/BUILD/repos/npm-cli/browse?at=refs%2Fheads%2Fcommander ) 

    The form would be magnolia [subcommand] [options]

    The @magnolia/cli package would install only one command, e.g. magnolia - or, should this be already taken by another package, it could be magnolia-cms. Then, from the terminal one could run something like

     

    $ magnolia setup -l light-modules -m 5.4.6
$ magnolia page -p light-modules/myModule myHome
$ magnolia component imageText -a myModule:pages/myHome@main
$ magnolia component imageText -g myModule:pages/myHome@footer
$ magnolia availability mtk:components/html -a myModule:pages/myHome@main
$ magnolia lightmodule -p light-modules/ myModule


    The advantage of this approach in my view is to have all sub-commands under one easy to remember "main" command magnolia which would make it easy to discover available subcommands by entering magnolia -h or  simply magnolia (without any arguments). Subcommands would also prevent clashes with same name commands other npm packages might possibly install.

    1. Michael Mühlebach

      Do we really want the command be named magnolia .... its just that this command might be used a lot and mgnl would be so nicer to type (this is what I always think when typing in those JIRA issue ids) 

      1. Magnolia International

        I would suggest ld

        The L and D keys are under One finger of your two hands, it takes <50ms to type them, whereas other solutions are closer to the second and error prone (big grin)

  8. Christopher Zimmermann

    I agree 100% that there should be one command. 

  9. Tomáš Gregovský

    relevant to suggestion in comment on  DEV-193 - Getting issue details... STATUS  when after npm install of cli you will need generate cli project specific files like packege.json, prototypes, etc. then to have same command for all stuff will be nicer, and of course shorter better (wink) e.g:

    npm install @magnolia/cli -g
    cd <workingDirectory>
    mgnl cli-setup
    mgnl create-light-module abc
    mgnl create-page abc
    ...

    1. Maxime Michel

      I don't get the "mgnl cli-setup" part. If this is a global tool, its mechanics shouldn't be replicated in N folders on your system. I'd see more something like "mgnl new ." to create the folder hierarchy, in which we wouldn't see any CLI code. Unless there is a reason for it (custom prototypes could be one, but seems advanced for the moment), this code doesn't need to populate the project hierarchy in my opinion. The tool dictates how to organize your project/business files, you adapt to it, and you get the promised result. The concerns are separated, updates are easier, and it's less confusing for users.

      1. Jan Haderka

        Actually, you might have different working environments and also different folders where your projects might lie. This is typical for the partners working for multiple clients. They then need to have mgnl/cli installed globally, so they can call same scripts everywhere, but at the same time they need to be able to quickly switch from working with one environment to another and be able to instrument those different instances of Magnolia in each respective environment.

        However, as you say yourself, this is advanced topic  so unless we find that we can't live without it when looking at the better install way (while working on DEV-193), we should ground this discussion.

        What I take from the discussion above is:

        • that we definitively want to go for having just one command magnolia or mgnl or similar installed and
        • that such script will then call all of the above agreed functionality for 
          • creation of the light module, 
          • creation of pages and components and 
          • for setting components as being available or autogenerated. 

        Let's stick to that for now and consider adding additional commands in the next version.

        1. Maxime Michel

          I'm not arguing against being able to run multiple projects on the same machine. But simply saying that IMHO they should be running as much common code as possible. The gulpfile would be the place to do project-specific tweaks/tasks to me, as that file is meant to be tweaked by end users, and there are snippets for every need all over the Internet. People do what they want to do with it (swap it with Grunt even), and we don't have to support anything.

          Now when it comes to having multiple instances on the same machine, I can see how it could be needed, but as you say it's advanced, so I would keep it out of here for now. I still wouldn't call it "cli-setup", though, but rather something like "mgnl new . --with-instance".

          1. Tomáš Gregovský

            there are custom nodes/properties in packege.json which are relevant for each specific project and NEED to be edited be user, e.g. dependency to front end tools such as"jquery", "angular", "bootstrap"... then reuse of resources provided by them "copyResources" section in current cli package.json, what jars (maven modules) has to be downloaded and installed, same for which version of magnolia, plus what has to be edited in magnolia.properties files. 

            This is something what I definitively need to configure when working on new project, and want to do it, from my <workingDirectory>, and instead of copypasting these lines from documentation, I would prefer to have command which will create it.

            1. Maxime Michel

              Well, as we discussed last week:

              • yes, a project package.json should be automatically created by us, in which we'd say what dependencies we require, but I would never expect "cli-setup" to create that for me, to me that name sounds really mysterious in what it's going to do.
              • on using a specific Magnolia version, isn't is reasonable to simply get the latest for the time being? If you're going to do light development, I don't see the point in not using the most current version.
              • the part about custom JARs, while nice and useful, is it something we want to provide with 1.0?

              In other words, I say we keep it as simple and stupid for now. But perhaps we need somebody in PM to define what needs to be in there?

              1. Tomáš Gregovský

                "on using a specific Magnolia version, isn't is reasonable to simply get the latest for the time being? If you're going to do light development, I don't see the point in not using the most current version."

                Yes for version number (well still discussable...) but what about CE, EESt., EEPro? empty, or bundle with demo? or just without demo?

                 

                "... "cli-setup" to create that for me, to me that name sounds really mysterious in what it's going to do." 

                Nobody says this is official name/keyword... I just used it for example...

      2. Federico Grilli

        I see the {{setup}} part as just an option especially for devs new to Magnolia to get a light dev environment up and running quickly w/o the need to know what to download, what to configure, etc. You can use it once, try things out and then forget about it. Like all (sub)commands of this CLI tool, you don't need to use all of them in a fixed order.