pyconx:	$(LIBRARY) $(LDLIBRARY)
		$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \
			$(UNICONOBJS) \
			$(BLDLIBRARY) $(LIBS) $(MODLIBS)  -Wl,-E -L../unicon/unicon/bin  -L/usr/lib -L/usr/X11/lib -L/usr/X11R6/lib  -lGLU -lGL -ljpeg -lXpm -lX11 -lz -lm -lgdbm -ltp -lnsl $(SYSLIBS) $(LDLAST)

7/2/04 - My usual goal when modifying Unicon is to figure out how to re-use and generalize existing pieces of the VM and runtime system. For example, to create a co-expression which invokes Py_Main(), I could write a whole bunch of new code, or I could look into existing code to do the trick. This capability might be viewed as a logical extension of the load() function which loads other Unicon programs into a co-expression. If load() is given a .py file instead of an icode file, it can go do the new thing. The low-level C co-expression creation code is a starting point, but for this co-expression to be well-behaved (especially upon termination) it sort of needs to look like an invocation on a Unicon procedure. The load() function has the VM instructions to invoke a procedure, but since its not Unicon VM, but C code that we are calling, we need to create a Unicon function which calls our C code. But this is what makefunc() in fload.r does for the function loadfunc() which loads C functions from shared libraries. So now it becomes a matter of assembling a few extensions to load(). I think.

7/15/04 - Unfortunately, we don't have a single-button identification of exactly what to get and install that is guaranteed to have the shared memory and/or pthreads features that are required.

7/16/04 - RRC is Redhat-centric, but the shared memory stuff did not run on my default/stock Redhat 9 box; I don't know exactly what config steps are needed to get it running on RH9.

7/17/04 - SUSE 9.1 rpm's are said to live at: ftp://ftp.suse.com/pub/suse/i386/9.1/suse/i586/. Trying to get YaST to suck in from this location seems to require backing up a directory or two. YaST sucked in gcc et al, apparently OK.

7/18/04 - Richie's sharedmem demo program works on SUSE 9.1 without any hassles, so I finally have a machine that can run it. Looking to add sharedmem interface to Unicon.

7/20/04 - Bought a machine on which to do this project without having to upgrade my main machine. It is an Athlon64 laptop, plan to install both 32- and 64-bit linuxen, downloading Fedora CD images. Unicon-monitoring-Python starts with Unicon-calling-Python capability, which I outline/prototype in callpython.c.

7/21/04 - Work on loading/calling python from Unicon is moving along. Compiles and links, next step is to try it out on a .py file and watch the cores dump. New laptop so far resisting Linux, repartitioning without reinstalling Linux doesn't work out of the box for NTFS/HPFS.

7/22/04 - Co-expressions are apparently not compatible with pthreads. After a co-expression switch, standard library functions tend to coredump in libpthread.so. Going to think about this, and switch back to working on a Unicon sharedmem interface. Might want it to look the same as the Python interface, need to see what that API is to look like. Maybe same as an underlying C++ API, eh?

7/23/04 - One option for "fixing" the co-expressions problem is to use pthreads to implement co-expressions. I have an earlier implementation of co-expressions using Solaris threads and semaphores... The only downside is that previous experience was that (Solaris) threads are a lot slower than co-expressions. This will end up reinforcing the asynchronous monitor model which we have already been discussing for performance reasons.

7/26/04 - Rich is TDY at NMSU. Rich is going to focus on producing the information, Clint is going to focus on consuming the information. First thing is logfile, then socket, then shmem (then coroutine if we get around to it).

Some time was spent discussing relative cost of coroutines versus threads. Unicon coroutines proved much much faster than Solaris lightweight threads in earlier tests; no signalling/semaphores needed, that was bundled into the switch itself. For monitors that do not need to inspect target program state, shmem is both adequate and probably better than a coroutine switch.

Might want mixture of monitors written in Python and in Unicon; Python might read information "natively", inspect the state of the interpreter in shmem, etc. Unicon might offer preexisting visualization tools and higher level graphics capabilities.

Richie has to remember to return a Key on Friday.

7/28/04 - NMSU COG was nice enough to enable the kernel features required for shm_open(), so now NMSU's slackware-based Linux runs farther than before, and dies on the same missing features (pthread_mutexattr_setpshared) as does my laptop running stock Redhat 9 without all the updates. It is another standard POSIX function, like the shm_open that was also a problem. Some platforms support pthread_mutexattr_setpshared partially (only for threads within a process), but we need the whole enchilada.

I've added access to Richie's shared memory facilities from Icon/Unicon. It looks like a variant of the built-in pipe, with an "s" appended to the extension, e.g. open(filename, "pw"). Next step will be some Python/Unicon shared memory interoperation testing with Rich.

I've also nearly completed a preliminary Alamo-compatibility library for using existing Icon visualization tools on events fed via logfile/socket/ sharedmem.

7/30/04 - went ahead and got shared memory live Python feed to Alamo last night. Next steps:

• cleanup / robustify what we've got so far
• add new kinds of events, such as list events
• measure performance, tune it. For example, Clint and Richie think that instrumenting call/return from C may well be 5-10X faster than instrumenting using Python (building strings, etc).
• plan a visualization tool to fit RRC-specific goals.

8/2/04 - Richie and Derek want LOW OVERHEAD. Writing the instrumentation in C to avoid Python overhead will be very revealing. If it still costs too much after that, there are several directions we can go to further reduce the cost. Sampling (and batching, for events that depend on context) events is obvious. Tuning the sharedmem facility, or looking for a simpler mechanism that can meet our communication needs, are also options. Often, the rule of thumb is: the faster the monitoring mechanism, the more intrusive it will be, and/or the lower-level the information it will provide. Also: first you build the access to the information that you actually want, then you figure out how to make it cheap. So, how do we measure "WHERE IS THE CPU GOING"? Add clock tick events, or add timer calls (maybe to the high-resolution timer) to measure time between other events. We identify the categories, and boundaries between the categories, of time we consider interesting. We select a granularity (or develop a scalable granularity) knowing that the more precise the measurement, the more it will cost.

Things Clint is thinking about:

• Add Unicon configuration code so that we can do a "make X-Configure name=intel_linux_rrc" and have the shared memory enabled. In order for this to work, it would help if Unicon could find libshm.so without writing its location in by hand. Options: (1) look for it via autoconf in a standard location or two, or (2) require Richie to copy it in to unicon/bin before building, and hardwire makefile to look there. Any other suggestions?
• Create a name for our project, perhaps something better than "Python Hooks". What we are really designing is a new generation UltraLight Monitoring Architecture== Ulma, a successor and replacement for Alamo. Its characteristics include: multilingual (can monitor Python, Unicon, C, etc...), hybrid instrumentation modes (logging, socket, pipe, shared mem...), hybrid control (synchronous, asynchronous, sampled...).
• Define a general-purpose (C-language) instrumentation primitive (macro or function) for insertion into the Python VM. Same instrumentation can select at runtime to write to logfile, socket, sharedmem,...
• On the receiving end, unify support for several kinds of event sources, so a monitor can read from any input source you give it.
• Think about how to live without being able to use pointers, or think about how to deliver pointer functionality through shared memory. Think about which kinds of events depend on pointers, and how they use them. Example: a list visualization tool receives an event which says that a list of 124 elements has been created, and the list visualization tool wants to show the contents of those 124 elements. Alamo solution: pass the list reference -- the monitor can use subscript operations to read that list just as easily as the target program. Ulma options: pass the entire list (as a string?) instead of a reference? pass a "handle" (object) that the monitor can use to access the list elements via proxy, and provide a sharedmemory means of performing proxy accesses.
• Think about how to provide inspection capability in an asynchronous (i.e. ultralight, the target program keeps running) event model. Copy-on-write? Sounds too expensive. Force monitor to specify what information they are going to want for each kind of event, up front at event-get time. For example, instead of just saying: "I want list creation events", be forced to say "I want list creation events, and report their size and element type", or "I want list creation events, with a full (deep) copy available for the monitor".

8/4/04 - the (grad) student hire is important. We need it settled because it affects what I do with my other grant hires this fall.

The obvious way to solve the configuration problem of having to copy in the path to libshm.so by hand, is to use a relative path.

8/6/04 - spent some of today working on improving the "packaging" of the Python->Unicon shared memory event demo .zip file, working on a .zip that will build everything out of the box. At the moment I'm building libshm and then configuring and building Unicon, and things mostly work except they require an addition to the LD_LIBRARY_PATH; I'd prefer that this be set in the Makefile itself so the build not require .cshrc modifications.

8/9/04 - status: we want a robust, portable implementation which will build out of the box on multiple platforms. The sharedmem stuff is is bigger and less portable than I expected. My current ULMA.zip builds libshm.so and then builds Unicon with sharedmem enabled on Solaris, with LD_LIBRARY_PATH set manually after unzipping. My next step is reexamine the source code for fsys.r and see if there is a code problem if libz is not found.

I checked about the libz thing, and sure enough the non-libz-code was badly wrong in one place in the Unicon runtime system.

Portability Status

Welcome to the Friday the 13th Edition of ClintsLog

• Level 0 (in Icon) was an all-or-nothing #ifdef that wrote to a file (disks are slow, by the way, and most monitors don't need all event types).
• Level 1 (early Alamo) is bit vector test filtering at runtime. Reduces cost of unused instrumentation to an if plus memory references. "Reduces" cost of used instrumentation to a co-expression switch. (Cost of co-expression switch not as cheap as I would like).
• Level 2 (later Alamo) has two copies of each instrumented function, with and without the #ifdef's. This further reduces cost of unused instrumentation, except in key functions (such as interp() ) that have many different kinds of events. Event masks (bit vector tests: memory references plus an "if") are still enabled (everywhere) because of those latter cases.
• Level 3 (ULMA) == ??. Want to eliminate the bit vector tests. If we know an event is going to be taken, just call its handler function. If we do not know, our "if" can just check if its handler function is NULL or not.

Edit Me