gr8conf eu 2013 speed up your development: groovyserv and grails improx plugin

Speed Up Your Development:GroovyServ & Grails Improx Plugin

@nobeansYasuharu NAKANO

GroovyServ

In case of Developing

a Groovy Script

Write Run

Common Strategy:“Trial and Error”

$ time groovy -‐e “println ‘Hello, Groovy’”Hello, Groovy

real 0m0.823suser 0m1.016ssys 0m0.096s

But, It takes 1 sec!

N timesTrial and Error

takes N sec.

“Many a little makes a mickle”

Waiting timemakes me

Forget nextaction

Why so slow?• JVM initialization• Loading Many Many Classes• .... from Disk

http://en.wikipedia.org/wiki/Java_performance#Startup_time

“ It seems that much of the startup time is due to IO-bound operations rather than JVM initialization or

class loading (the rt.jar class data file alone is 40 MB and the JVM must seek a lot of data in this huge file).

Doom of JVM Languages?

Any solutions?

$ groovyConsole

DemogroovyConsole

groovyConsole• Handy for trying Java/Groovy APIs, etc.• But features as editor aren’t so rich:

• No code completion• No syntax checking• No auto import• etc.

• I want to use my favorite IDE/editor!

Other solutions?

GroovyServ“GroovyServ makes Groovy’s startup time much

faster, by pre-invoking Groovy as a server.”

http://kobo.github.io/groovyserv/

DemoGroovy and GroovyServ

$ time groovyclient -‐e “println ‘Hello, GroovyServ’”Hello, GroovyServ


Normal Groovy:real 0m0.823suser 0m1.016ssys 0m0.096s

Normal Groovy User Groovy

Script

Shell Environment Java VM

File System

CLASSPATH

cmd line args

stdin

stdou

t

stder

r

Ctrl-Cother ENVs

exit

statu

s

Syste

m.in

Syste

m.out

Syste

m.err

groovy

User GroovyScript

Shell Environment

Java VM

File System

User GroovyScriptCLASSPATH

cmd line args

stdin

stdou

t

stder

r

Ctrl-C

other ENVs

≪file≫Authtoken

exit

statu

s

TCP/IP

Syste

m.in

Syste

m.out

Syste

m.err

groovyservergroovyclient

GroovyServ


a Groovy Script

GroovyServ Way• Write code with your favorite IDE/editor• Run the code by using GroovyServ as

External Tool• You can also see the result on a console

of IDE

DemoGroovyServ way

Available Editors/IDEs• IntelliJ IDEA• Eclipse/STS/GGTS• Sublime Text 2• Vim + QuickRun.vim• and more...

GroovyServ“GroovyServ makes Groovy’s startup time much

faster, by pre-invoking Groovy as a server.”

http://kobo.github.io/groovyserv/

How to Install

How to Install• Downloadable Binaries for

• Mac OS X (built at Mac OS X 10.8 (x86_64))• Windows

• bundled in Groovy Windows Installer• (built at WindowsXP (32bit) (x86))• (built at Windows7 (64bit) (AMD64))

• Linux (i386) (linked with glibc 2.9)• Self-build (required Gradle)• Homebrew (only Mac)

• $ brew install groovyserv

• http://kobo.github.io/groovyserv/howtoinstall.html

No setting required• If there is a binary set, you can use it:• $ /xxx/groovyserv/bin/groovyclient

• Adding to PATH, you can easily use it:• $ groovyclient

How to Use

Commands• groovyclient• groovyserver

groovyclient

• Invokes a groovy file or one-liner code, by sending a request to server

• Also invokes groovyserver process automatically if not exists

groovyserver

• Invokes a JVM process as a server.• Should be used only when you want to

specify special options.• e.g.• -‐v verbose option

• -‐-‐allow-‐from for remote feature

Available for Other JVM Languages

http://nobeans-en.blogspot.jp/2011/07/high-speed-start-up-jythonclojure-by.html

Jython$ alias gythonserver="env CLASSPATH=/your/jython.jar groovyserver"$ alias gython="groovyclient -‐e 'import org.python.util.jython; jython.main(args)' -‐-‐"

$ gythonserver -‐r...

$ time gython -‐c "print('Hello')"Hello


Clojure$ alias glojureserver="env CLASSPATH=/your/clojure.jar groovyserver"$ alias glojure="groovyclient -‐e 'import clojure.main;main.main(args)' -‐-‐"

$ glojureserver -‐r...

$ time glojure -‐e "(println 'Hello)"Hello

real 0m0.053suser 0m0.001ssys 0m0.004s main method of

closure.main class

• A script to get schedule data from groupware

• As editor’s macro• As mail filter• etc.

Using at Runtime

Other Features• Access Control• Propagation of CLASSPATH• Propagation of Environment Variables• Handling System#exit()• Dynamic CWD

Improx Plugincodenamed:GrailsServ


Grails Application

Grails’ Test supports is Great.

How to run tests• All tests• grails test-‐app

While I’m writing a test,I want to run only it.

I want to run the test as fast as possible.

How to run a specified test fast• Run only a unit test• test-‐app unit: <FQCN of TestClass>

• Run only a integration test• test-‐app integration: <FQCN of TestClass>

• Run only a functional test• test-‐app functional: <FQCN of TestClass>

Why does it needa test phase?

If test phase not specified when run unit test, integration phase is also run.

$ cat test/unit/test/app/sample/FooTests.groovy....class FooTests { void testSomething() { // do nothing }}

$ time grails test-‐app FooTests| Packaging Grails application.....| Tests PASSED -‐ view reports in /xxx/test-‐reportsreal 0m15.375suser 0m35.119ssys 0m1.858s

Oops!�� Integration�� phase�� running!?

$ time grails test-‐app unit: FooTests| Completed 1 unit test, 0 failed in 546ms| Tests PASSED -‐ view reports in /xxx/test-‐reports


Less�� than�� a�� half�� of�� previous�� sample!

Specifying a phase can avoidrunning an extra phase

Without a phase:real 0m15.375suser 0m35.119ssys 0m1.858s

Required to run test-app• FQCN of target test class• Test phase of target test class

By the way

How do you run a test while writing it?• (A) By IDE support (GGTS, IntelliJ, etc.)• (B) By “grails test-app” as standalone• (C) By “grails test-app” via Interactive

Mode• (D) Others

Interactive Mode• Very very useful• Makes command execution faster because

the JVM doesn't have to be restarted for each command

• $ grails

• TAB completion is available• Since Grails 2.0

DEMOInteractiveMode/Pros

But still there is trouble• It needs frequent switching of windows

for use while writing code in IDE/editor• To run a test, you must prepare:

• FQCN of target test class• Test phase of target test class

DEMOInteractiveMode/Cons

I’m too tired from such Switching,

Copy & Paste, etc...

Improx PluginProvides the way of using interactive mode

from other process via TCP.

http://grails.org/plugin/improx

DEMOImprox Plugin

Improx Plugin• You can run a test

• Faster than general IDE supports because using interactive mode

• Directly from IDE/editor without complicated operations

Available Editors/IDEs• IntelliJ IDEA• Eclipse/STS/GGTS• Sublime Text 2• Vim + QuickRun.vim• and more...

InteractiveModePROXy

Grails

Improx

Shell Environment

≪script≫improxClient

TCP/IP

≪script≫improxSmartInvoker

≪class≫

InteractiveMode≪class≫

GrailsConsole

≪command≫

improx-start

≪command≫

improx-stop

≪command≫

improx-install-resourcesother clients

≪class≫

GrailsScriptRunner

echo back deleg

ate

run script

start

stop

grails command

expand client scripts

use

• As same as interactive mode• install-‐plugin

• uninstall-‐plugin

• Special built-in commands• create-‐app

• quit

• stop-‐app• exit

• !(shell command)

• Using threads complexly• run-‐app

Unsupported Commands

How to Install

BuildConfig.groovy:

plugins {//... compile ":improx:0.2"//...}

Install Plugin

Install Client Scripts$ grails improx-‐install-‐resources| Improx resources installed successfully: /xxx/improx-‐resources

$ chmod +x improx-‐resources/scripts/*.sh$ mv improx-‐resources $HOME/.improx-‐resources

If�� once�� you�� store�� them�� to�� global�� path,�� you�� don't�� need�� the�� preparation��

for�� other�� projects.

How to Use

Grails Commands• improx-‐start

• improx-‐stop

• improx-‐install-‐resources

improx-‐start / improx-‐stop

• improx-‐start

• Starts an interactive mode proxy server• improx-‐stop

• Stops the interactive mode proxy server which is running

improx-‐install-‐resources

• Installs “improx-resources” directory which has some client scripts into your application project

improx-‐resources/ !"" scripts #"" improxClient.groovy #"" improxClient.sh #"" improxSmartInvoker.groovy !"" improxSmartInvoker.sh

Client Scripts• improxClient.sh• improxClient.groovy• improxSmartInvoker.sh• improxSmartInvoker.groovy

improxClient.(sh|groovy) <Command>

• Runs a command in interactive mode via TCP port of improx server

• Very simple

$ improxClient.sh helpExecuting 'help' via interactive mode proxy.........Usage (optionals marked with *):grails [environment]* [options]* [target] [arguments]*

...(snipped)...

test-‐apptomcatuninstall-‐pluginupgradewarwrapper$

improxSmartInvoker.(sh|groovy) FILE_PATH

• Invokes any *.groovy file in an appropriate way

• Very useful for IDE/editor

How SMART is it?

grails> test-‐app unit: sample.SampleUnitTests

$ improxSmartInvoker.sh /path/to/yourApp/test/unit/sample/SampleUnitTests.groovy

Runs the following commandon the interactive mode:

grails> test-‐app integration: sample.SampleIntegTests

$ improxSmartInvoker.sh /path/to/yourApp/test/integration/sample/SampleIntegTests.groovy

Runs the following commandon the interactive mode:

$ improxSmartInvoker.sh /path/to/yourApp/test/functional/sample/SampleFuncTests.groovy

grails test-‐app functional: sample.SampleFuncTests

Because�� functional�� test�� doesn’t�� work�� well�� on�� interactive�� mode�� at�� present.

Runs the following commandas new standalone Grails process:

$ improxSmartInvoker.sh /path/to/scriptDir/myTribialScript.groovy

groovy /path/to/scriptDir/myTribialScript.groovy

If�� you've�� installed�� GroovyServ,�� the�� groovyclient�� is�� automatically�� used�� instead�� of�� groovy�� command.

Runs the following commandas a normal Groovy script:

All you have to do isto set

improxSmartInvokeras External Tool

of IDE/editorhttp://kobo.github.io/grails-improx/guide/integrationWithEditors.html

SimpleProtocol

TCP• Request:

COMMAND

• Whole response is result of the command• You can also use as client

• e.g. telnet, nc

$ telnet localhost 8096Trying ::1...Connected to localhost.Escape character is '^]'.help......Usage (optionals marked with *):grails [environment]* [options]* [target] [arguments]*

...(snipped)...

uninstall-‐pluginupgradewarwrapper$

HTTP• Request

GET /COMMAND HTTP/[0-‐9.]+

• Whole response is result of the command• You can also use as client

• e.g. Web browser, wget, curl

• GroovyServ

• http://kobo.github.io/groovyserv/index.html

• https://github.com/kobo/groovyserv

• Improx - Interactive Mode Proxy

• http://kobo.github.io/grails-improx/guide/

• http://grails.org/plugin/improx

• https://github.com/kobo/grails-improx

• Demo code

• https://github.com/nobeans/gr8confeu2013-demo

Appendix

GroovyServUnder the hood

Under the hood• Client-Server Communication• Access Control• Propagation of CLASSPATH• Propagation of Environment Variables• Handling System#exit()• Dynamic CWD

Client-Server Communication

Transfer of Standard In/Out/Err Stream

$ groovyclient -‐e \"println ‘Hello, GroovyServ’"

groovyclient

ClientConnection

OutputStream

≪script≫

println “Hello, GroovyServ!”

≪System.out≫

DynamicDelegatedPrintStream

writes data converted based on a protocol into output stream of socket

Hello, GroovyServ!

if Channel header == ”err”To STDERR

if Channel header == ”out”To STDOUT

Client console

$ groovyclient -‐e \"System.in.eachLine{ println it*5 }"

ClientConnection

PipedOutputStream

PipedInputStream

groovyclient ≪thread≫

StreamRequestHandler

≪script≫

System.in.eachLine { ... }

Connected

rawData = Socket.inputStream.read()bodyData = convert(rawData)pipedOutputStream.write(bodyData)

≪System.in≫

DynamicDelegatedInputStream

Protocol over TCP

InvocationRequest'Cwd:' <cwd> LF'Arg:' <arg1> LF'Arg:' <arg2> LF :'Env:' <env1>=<value1> LF'Env:' <env2>=<value2> LF :'Cp:' <classpath> LF'AuthToken:' <authToken> LFLF

where: <cwd> is current working directory. <arg1>,<arg2>.. are commandline arguments which must be encoded by Base64. (optional) <env1>,<env2>.. are environment variable names which sent to the server. (optional) <value1>,<valeu2>.. are environment variable values which sent to the server. (optional) <classpath> is the value of environment variable CLASSPATH. (optional) <authToken> is authentication value which certify client is the user who invoked the server. LF is line feed (0x0a, '\n').

StreamRequest

'Size:' <size> LFLF<body from STDIN>

where: <size> is the size of body to send to server. <size>==-1 means client exited. <body from STDIN> is byte sequence from standard input.

InvocationResponse

'Status:' <status> LF

where: <status> is exit status of invoked groovy script.

StreamResponse

'Channel:' <id> LF'Size:' <size> LFLF<body for STDERR/STDOUT>

where: <id> is 'out' or 'err', where 'out' means standard output of the program. 'err' means standard error of the program. <size> is the size of chunk. <body from STDERR/STDOUT> is byte sequence from standard output/error.

Access Control

Against Evil Request

From Evil Client

Client Address Checking / Authtoken• Request only from loopback address is

allowed.• Authtoken is generated and stored into a

file when server process starts.• $HOME/.groovy/groovyserver/authtoken-‐<PORT>

• Authtoken always is used for checking that the request is from valid owner user.

-‐-‐allow-‐from

• By default, A request only from loop-back address is available.

• You can allow specified addresses.$ groovyserver -‐-‐allow-‐from 192.168.1.1

SECURITY RISK: Be careful when using this option

-‐-‐authtoken

• By default, authtoken is generated automatically.

• You can specify any string.

SECURITY RISK: Be careful when using this option

$ groovyserver -‐-‐authtoken MY_AUTHTOKEN

At your own risk

Propagation of CLASSPATH

Client and Server are Different

Process

≪process≫groovyserver

≪process≫groovyclient

CLASSPATH=/my/classpath/xxx.jar

CLASSPATH=/my/classpath/zzz.jar

Individual Env for each

If GroovyServ were too simple, ...



CLASSPATH=/my/classpath/foo.jar



Please execute this!

new foo.Foo().execute()





I don’t know its class!






I don’t know its class!

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:script_from_command_line: 1: unable to resolve class foo.Foo @ line 1, column 1. new foo.Foo().execute() ^

1 error



But GroovyServ is enough smart




You got it!




CLASSPATHwhich is propagated

from client is Volatile




OK





Done



CLASSPATH=/my/classpath/bar.jar


new bar.Bar().execute()


OK


Done

How to specify CLASSPATH from

Client

-classpath / -cp / --classpath

$ groovyclient -‐cp /my/classpath/foo.jar -‐e “new foo.Foo().execute()”

CLASSPATH env$ export CLASSPATH=/my/classpath/foo.jar$ groovyclient -‐e “new foo.Foo().execute()”

Default CLASSPATHon Server

CLASSPATH env$ export CLASSPATH=/my/classpath/foo.jar$ groovyserver


I feel some power...








I already know foo.jar!


Done






new bar.Bar().execute()


OK



Done


Propagation of Environment

Variables

Handywith IDE/editor


≪process≫groovyclientCLASSPATH=

/my/classpath/xxx.jar

How to propagate env

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1

KEY_B=VALUE_B

KEY_A_2=VALUE_A_2

groovyclient options• -‐Cenv <substr>

• Passes environment variables of which a name includes specified substr

• -‐Cenv-‐all

• pass all environment variables• -‐Cenv-‐exclude <substr>

• don't pass environment variables of which a name includes specified substr




-‐Cenv _A_

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1

KEY_B=VALUE_B

KEY_A_2=VALUE_A_2




-‐Cenv-‐all

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1

KEY_B=VALUE_B

KEY_A_2=VALUE_A_2




-‐Cenv-‐all -‐Cenv-‐exclude _2

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1

KEY_B=VALUE_B

KEY_A_2=VALUE_A_2

Caution:ENV on server isn’t cleared after each

invocation




-‐Cenv KEY_A_1

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1




-‐Cenv KEY_A_2

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2




-‐Cenv KEY_B

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

KEY_A_1=VALUE_A_1

KEY_A_2=VALUE_A_2

KEY_B=VALUE_B

Handling System#exit()

Server processmust not be terminated,

even if a script executes System.exit()

SecurityManagerpublic class NoExitSecurityManager2 extends NoExitSecurityManager {

public void checkExit(final int code) { throw new SystemExitException(code, "called System.exit(" + code + ")"); }

}

GroovyMainprivate boolean run() { try { //... processArgs(...) //... } catch (SystemExitException e) throw e; } catch (Throwable e) { //... }}

static void processArgs(...) { //... if (!process(cmd, classpath)) { // Disabled because this causes a secondary disaster //System.exit(1); } //...}

Dynamic CWD

CWD =Current Working

Directory

$ cd /tmp$ groovyserver -r

$ cd /home/nobeans$ cat test.txtCan you read me?

$ groovyclient -e ‘println(new File(“test.txt”).text)’

Caught: java.io.FileNotFoundException: test.txt (No such file or directory) ...SNIP...


CWD=/tmp

I wanna use “test.txt”of current directory.

Is it “/tmp/test.txt” ?Hmm, such a file not found.

CWD of client should be set to server

for each invocation.But, ...

In Java,CWD is thoroughly

hidden

Solution• Setting to “user.dir” system property

• It affects File#getAbsolutePath()• Changing CWD of native JVM process

• Many classes don’t use “user.dir” to complete an absolute path.

• But, JNI is troublesome...

JNA is easy way to use native API

UnixLibC.groovy:interface UnixLibC extends com.sun.jna.Library.Library { int chdir(String dir) int setenv(String envVarName, String envVarValue, int overwrite)}

pom.xml: <dependency> <groupId>net.java.dev.jna</groupId> <artifactId>jna</artifactId> <version>3.2.2</version> </dependency>

Usage:def libc = com.sun.jna.Native.loadLibrary("c", UnixLibC.class)libc.chdir(“/tmp/foo/bar”)

$ cd /tmp$ groovyserver -r

$ cd /home/nobeans$ cat test.txtCan you read me?

$ groovyclient -e ‘println(new File(“test.txt”).text)’

Can you read me?


CWD=/tmp

I wanna use “test.txt”of current directory.


CWD=/home/nobeans

Is it “/home/nobeans/test.txt” ?OK, I got it.

Yes!

gr8conf eu 2013 speed up your development: groovyserv and grails improx plugin

Technology

suser 0m0

ssys 0m0

unit test testapp unit

time grails testapp

needa test phase

specified test fast

groovyreal 0m0

groovyservreal 0m0