1. D2082 Runtime Replace Hw.ncpu With Hw.logical Cpu For Machine Learning

This is a continuation of the previous post. I decided to give the section pertaining to Common Lisp its own post as I think there is some good educational value in this part itself.

We carry on from where we left off last time. We continue with the same squaring number callback example. As a quick refresher, the idea is to implement a synchronous callback scenario.

The function client invokes another function squarify which squares the passed value and invokes a callback function callback. How it’s done in Common Lisp Let’s start off with our first attempt to implement the solution in Common Lisp.;;;; Callback demo using the squarify example. (defpackage:callback-demo-user (:use:cl)) (in-package:callback-demo-user) (defun callback(n) (format t 'Received: d%' n)) (defun squarify(n cb) (funcall cb (. n n))) (defun client (let ((n (progn (princ 'Enter a number: ') (read)))) (squarify n #'callback))) CALLBACK-DEMO-USER (client) Enter a number: 19 Received: 361 NIL That’s the direct equivalent of all the demos shown so far. However, since Common Lisp is a functional language (albeit not as pure as, say, Scheme or Haskell), we can certainly do better! In most Functional Programming languages, higher order functions are usually deployed to do the job.

So let’s see if we can cook up something nicely functional like. Here’s a first attempt: (defun client (funcall #'(lambda (n) (format t 'Received: d%' n)) (funcall #'(lambda (n) (. n n)) (funcall #'(lambda (princ 'Enter number: ') (read)))))) Which produces: CALLBACK-DEMO-USER (client) Enter number: 19 Received: 361 NIL As expected! Now, as you may know, funcall simply takes a function and some arguments (optional), and applies the function to those arguments.

D2082 Runtime Replace Hw.ncpu With Hw.logical Cpu For Machine Learning

In this case, we simply compose them in the proper order so that the types match up: read a number - square it - print message. However, let’s work our way to a generic compose function that simulates the behaviour of Haskell’s composition operator. The previous function can be improved by defining a new version that composes the three functions in the mentioned order (so as to match types): The compose function: (defun compose (fn gn hn) #'(lambda (&rest args) (funcall fn (funcall gn (apply hn args))))) And the client to test it: (defun client (funcall (compose #'(lambda (x) (format t 'Received: d%' x)) #'(lambda (x) (. x x)) #'(lambda (princ 'Enter a number: ') (read))))) And the output is the same: CALLBACK-DEMO-USER (client) Enter a number: 19 Received: 361 NIL So what’s changed? Well, taking inspiration from the nested funcall function, we defined compose to invoke the functions in the proper order – first read the number, and then square it, and then finally print it! (Remember that the functions are composed in reverse order in which they are entered). Note that the last function invocation is done using apply instead of funcall because &rest args produces a list of arguments, and funcall does not work with that (unless the function definition takes a list itself as a parameter, but that is not the general case, and apply works very well with lists and destructures them correctly.

How can we make this generic enough though? We notice the pattern – we invoke apply on the innermost function call, but we use funcall for the rest of the function call chain. This means that we must handle two cases – if there is a single function passed in, we should simply use apply on that, and if not, we should take care to chain them up as discussed. This lends itself to a nice recursive definition as shown next. The updated compose function: (defun compose (&rest funcs) (labels ((f (funcs args) (if (null (cdr funcs)) (apply (car funcs) args) (funcall (car funcs) (f (cdr funcs) args))))) #'(lambda (&rest args) (f funcs args)))) ) And the test client for it: (defun client (funcall (compose #'(lambda (x) (format t 'Received: d%' x)) #'(lambda (x) (.

x x)) #'(lambda (princ 'Enter number: ') (read))))) And the output: CALLBACK-DEMO-USER (client) Enter number: 19 Received: 361 NIL Explanation: What we do is simply generalise the three-function version of compose into a generic function. For this we, define an internal function f that takes the supplied functions and the arguments as input. F then recursively decomposes the function applications. The base condition (stopping condition) is when there is only one function left. The (if (null (cdr funcs)) bit then takes care to return the only apply call that we need, and that is of course, applied to the args argument. As the recursion unwinds the call stack, successive funcallS are applied at each stage. This is exactly in line with the algorithm discussed at the end of the last section.

Now we are almost home and dry! Pay special attention to the order in which the lambda equivalents of the functions are entered in the client function. They are applied in the following order – callback, squarify, and then client. We could stop here, but there’s one more change that we can make. The current version of compose works absolutely as expected, but the intuitive order of supplying functions is the opposite of what we could expect as a user. The expected order would be, in English, “read in the number, square it, and then print out a message indicating that the number was received”.

Let’s fix that last bit for out final version of compose. Final version of compose:;;; final version of compose (defun compose(&rest funcs) (labels ((f (funcs args) (if (null (cdr funcs)) (apply (car funcs) args) (funcall (car funcs) (f (cdr funcs) args))))) #'(lambda (&rest args) (f (reverse funcs) args))))) And the corresponding test code:;;; test out the final version of compose (defun client (funcall (compose #'(lambda (princ 'Enter a number: ') (read)) #'(lambda (x) (. x x)) #'(lambda (x) (format t 'Received: d%' x))))) And now let’s test out and see if it works! CALLBACK-DEMO-USER (client) Enter a number: 19 Received: 361 NIL Success! The only difference is this line: (f (reverse funcs) args). We simply reverse the order of the received functions while passing it to the recursive function f, and the rest of the code remains exactly the same! And, of course, this is purely functional!

Sweet, ain’t it? The compose function could be optimised in multiple ways – converting it to an iterative version for instance, but conceptually, this works exactly as advertised. Conclusion This post illustrates why I love Common Lisp! Even as I make my journey through the world of Common Lisp, my admiration for it only grows. If there is some feature that we would like to incorporate into the language, it can be done in a just a few lines of code! No other language truly comes close in terms of expressiveness and extensibility.

Currently, we've got excellent code to run Cython in parallel as part of the Sage build process. However, once we ask distutils to begin compiling that code, everything is done in serial, because distutils works solely in serial.

The Code The attached file changes that. This (somewhat brutally) hacks distutils to dispatch the calls to build C/C extensions in parallel, using pyprocessing.

(I totally jacked William's code from for this.) Here's how to put this code in place:. download the attached file ( buildext.py). replace $SAGEROOT/local/lib/python2.5/distutils/command/buildext.py with the new version. Now, if you want to test the new code, do the following:.

set the environment variable SAGEPARALLELDIST to something. (The code just checks to see if the variable is defined at all.). set the environment variable MAKE to MAKE -j2, where 2 is replaced by the number of simultaneous build processes you want.

Notes If you want to test this, don't go around touching the.pyx files in the Sage library, since Cython is much slower than gcc. Instead, simply go around touching the.c and.cpp files in $SAGEROOT/devel/sage/sage. One of the cool features we added with the new build system is that these files get recompiled when they change. There is now a line that prints as part of the build process that looks like: Total time spent compiling C/C extensions: 5. So try touching a bunch of files in the Sage library, and seeing what kind of speedups you get. There are two caveats I want to offer with this code:.

Michael points out that numpy does a lot with distutils. I could very well have broken their use of distutils. I don't do anything involving dependency tracking between extensions. In particular, if there are extensions that have to be built in a certain order, this code could break it. (This code still compiles all source files within an extension in the usual way.) Neither Michael nor I could think of a situation where this would break anything in Sage, but who knows. The Plan So while this code is cool, and will definitely save a ton of CPU time on sage.math ( cough mabshoff cough), the plan is not to maintain it as a part of the Sage Python spkg indefinitely. Instead, if it seems to work well, then we should try to clean this code up a bit more and upstream it, since pyprocessing is standard in Python 2.6.

Just out of curiosity, did you try to get this to work without modifying buildext.py at all, but by simply changing stuff in that module at runtime, like I illustrate in? There is a slight chicken and egg problem here since we build pyprocessing after python, so if someone sets the wrong environment variable the build of Python itself will blow up. To fix this:. make sure to unset SAGEPARALLELDIST for the python.spkg as well as the pyprocessing.spkg. make all the other python based spkgs depend on pyprocessing, so it is immediately build after Python Then this should work out of the box, even though we should still test this with a couple rounds of builds of Sage with a high parallel level to make sure we do not hit any race conditions.

Cheers, Michael. Ok, new version of the code is up. This is done differently than last time; rather than patch our python, I'm simply inserting the code in our setup.py, and calling distutils with the new code only from Sage. So, to test this, do the following:. download and apply trac-4652-testing.patch.

set the env variable SAGEPARALLELDIST to anything. set the env variable MAKE to something like make -j168 (where 168 is the number of simultaneous threads you want). sage -br or sage -ba If this seems to work for people, then we can try building Sage from scratch with it. (I'm going to try setting my machine to do that overnight with 4.0.1.rc2, I think.) If it works fine, I just want to add a little bit of documentation and then I'm happy to have this merged. If it works for a release or so, I'll start working on getting something like this upstream.

(I have no idea how uphill that battle will be.).