Forum Discussion
Command line compilation time -- Can we multi-thread
Command line compilation time -- Can we multi-thread
We have integrated our Blaze Rules into our continuous integration process. This means we build our rules along with our Java code 5-6 times per day. The rules currently take 22 minutes to compile when compiled from the command line in a single threaded manner. We tried modifying our build to compile Blaze multi-threaded with little success. The process is several times slower than our single threaded build. Our current single threaded compilation process uses NdAdvisorProjectCompiler.main() and that is what we used in each of the threads. We're wondering if we're using the wrong approach.
private void compile(Service service) { getLog().debug( "Starting compilation of Blaze service." ); int i = 0;String[] args = new String[4]; args[i++] =
"-server" ; File f = new File(service.getServer());args[i++] = f.getPath();
args[i++] = "-name"; String serverOutName = stripSuffix(service.getServer(),
SERVER ); f = new File(serverOutName + COMPILED_SERVER );args[i++] = f.getPath();
getLog().debug( "Executing NdAdvisorProjectCompiler with args:" ); for ( int j = 0; j < args. length ; j++) { getLog().debug(args
}
NdAdvisorProjectCompiler.main(args);
getLog().debug( "Finished compiling Blaze project" );}
7 Replies
No, you can't multithread it. However, if your .server file contains multiple projects, then you could split it (just for compilation) into multiple .server files with one project in each and compile those in parallel.
- If we had 8 .server files then we'd spin off 8 threads, each using: NdAdvisorProjectCompiler.main()That doesn't give the performance we expect. Should we be using some other compilation method on NdAvisorProjectCompiler?
NdAdvisorProjectCompiler is the only compilation method that is out there. When running several instances of it from multiple threads you will get better performance if you have multiple CPU cores available on the build box. Just make sure you have enough memory available as well - compilation takes more memory than Blaze normally does.
Do you use compiled sequential mode in your project? If so, it will slow down compilation time considerably. In some cases you may want to run your project in Blaze IDE's profiler and disable compiled sequential mode for non-critical rulesets/functions that are used rarely and do not affect runtime performance much but do slow down the compilation process.
- I'm trying to do something similar, however if compilation fails there doesn't appear to be any real way to detect it other than looking at the logs. Is there a different method to call which returns a boolean or throws an exception when the pre-compile fails?
I think the compilation process normally throws an exception if it fails. Our code catches that.
One of our developers wrote a multi-process compilation process that farms off compilation to various worker processes. The compilation process is almost completely CPU bound so we run the same number of processes as there are cores in the machine. It doesn't speed up the compilation of a single rule base but does have a big impact on a set of rulebases.
- Multi-threaded compilation doesn't work because there are some single threaded places in the compilation code itself. You have to parallel compile using multiple processes instead of usign multiple threads in a single process.
As an example, if I have something like this...
try { System.out.println("Starting compile"); NdAdvisorProjectCompiler.main(args); System.out.println("Compile completed."); } catch(java.lang.Throwable t) { System.err.println("BUILD FAILED!"); t.printStackTrace(); }... if compilation fails, the 'Compile completed' print statement is never reached, nor is the catch block. It just exits.
